{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Zero Shot Learning - Cross Lingual Named Entity Recognition with XLM-Roberta\n",
    "\n",
    "## [Link to my Youtube Video Explaining this whole Notebook](https://www.youtube.com/watch?v=urMUa4Nw_B8&list=PLxqBkZuBynVQEvXfJpq3smfuKq3AiNW-N&index=7)\n",
    "\n",
    "[![Imgur](https://imgur.com/oOpkFLo.png)](https://www.youtube.com/watch?v=urMUa4Nw_B8&list=PLxqBkZuBynVQEvXfJpq3smfuKq3AiNW-N&index=7)\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First What is BERT?\n",
    "\n",
    "BERT stands for Bidirectional Encoder Representations from Transformers. The name itself gives us several clues to what BERT is all about.\n",
    "\n",
    "BERT architecture consists of several Transformer encoders stacked together. Each Transformer encoder encapsulates two sub-layers: a self-attention layer and a feed-forward layer.\n",
    "\n",
    "### There are two different BERT models:\n",
    "\n",
    "- BERT base, which is a BERT model consists of 12 layers of Transformer encoder, 12 attention heads, 768 hidden size, and 110M parameters.\n",
    "\n",
    "- BERT large, which is a BERT model consists of 24 layers of Transformer encoder,16 attention heads, 1024 hidden size, and 340 parameters.\n",
    "\n",
    "\n",
    "\n",
    "BERT Input and Output\n",
    "BERT model expects a sequence of tokens (words) as an input. In each sequence of tokens, there are two special tokens that BERT would expect as an input:\n",
    "\n",
    "- [CLS]: This is the first token of every sequence, which stands for classification token.\n",
    "- [SEP]: This is the token that makes BERT know which token belongs to which sequence. This special token is mainly important for a next sentence prediction task or question-answering task. If we only have one sequence, then this token will be appended to the end of the sequence.\n",
    "\n",
    "\n",
    "It is also important to note that the maximum size of tokens that can be fed into BERT model is 512. If the tokens in a sequence are less than 512, we can use padding to fill the unused token slots with [PAD] token. If the tokens in a sequence are longer than 512, then we need to do a truncation.\n",
    "\n",
    "And that’s all that BERT expects as input.\n",
    "\n",
    "BERT model then will output an embedding vector of size 768 in each of the tokens. We can use these vectors as an input for different kinds of NLP applications, whether it is text classification, next sentence prediction, Named-Entity-Recognition (NER), or question-answering.\n",
    "\n",
    "\n",
    "------------\n",
    "\n",
    "**For a text classification task**, we focus our attention on the embedding vector output from the special [CLS] token. This means that we’re going to use the embedding vector of size 768 from [CLS] token as an input for our classifier, which then will output a vector of size the number of classes in our classification task.\n",
    "\n",
    "-----------------------\n",
    "\n",
    "![Imgur](https://imgur.com/NpeB9vb.png)\n",
    "\n",
    "-------------------------"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In this Notebook, I will fine-tune XLM-R on the German corpus dataset and will see it yields an F1-score of around 85%\n",
    "\n",
    "### And then without any additional training or fine-tuning I will apply that same model on some other languages in our corpus and in those other languages I will achieve some modest performance.\n",
    "\n",
    "-----------\n",
    "\n",
    "\n",
    "### What’s New in XLM-Roberta?\n",
    "\n",
    "The Facebook AI team released XLM-RoBERTa in November 2019 as an update to their original XLM-100 model. Both are transformer based language models, both rely on the Masked Language Model objective and both are capable of processing text from 100 separate languages. The biggest update that XLM-Roberta offers over the original is a significantly increased amount of training data. The cleaned CommonCrawl data that it is trained on takes up a whopping 2.5tb of storage! It is several orders of magnitude larger than the Wiki-100 corpus that was used to train its predecessor and the scale-up is particularly noticeable in the lower resourced languages. The “RoBERTa” part comes from the fact that its training routine is the same as the monolingual RoBERTa model, specifically, that the sole training objective is the Masked Language Model. \n",
    "\n",
    "There is no Next Sentence Prediction like BERT or Sentence Order Prediction like ALBERT.\n",
    "\n",
    "-----------\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5Xx2U4kp3pUJ",
    "outputId": "8e432881-a0d7-4c04-f6b8-aa622d09cab8"
   },
   "outputs": [],
   "source": [
    "!pip install transformers datasets seqeval -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "from seqeval.metrics import f1_score\n",
    "from transformers import EvalPrediction\n",
    "\n",
    "from transformers import TrainingArguments\n",
    "from datasets import get_dataset_config_names\n",
    "from datasets import load_dataset\n",
    "\n",
    "from transformers import DataCollatorForTokenClassification\n",
    "\n",
    "from collections import defaultdict\n",
    "from datasets import DatasetDict\n",
    "\n",
    "from collections import Counter\n",
    "\n",
    "from transformers import Trainer\n",
    "\n",
    "import torch.nn as nn\n",
    "from matplotlib import pyplot as plt\n",
    "from transformers import XLMRobertaConfig\n",
    "from transformers.modeling_outputs import TokenClassifierOutput\n",
    "from transformers.models.roberta.modeling_roberta import RobertaModel\n",
    "from transformers.models.roberta.modeling_roberta import RobertaPreTrainedModel\n",
    "\n",
    "import torch\n",
    "\n",
    "from transformers import AutoConfig\n",
    "from transformers import AutoTokenizer\n",
    "\n",
    "from seqeval.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JcHMHgKP3pUR",
    "outputId": "0842ef57-8d63-4e06-f5ae-1945cd13c880"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "XTREME has 183 configurations\n"
     ]
    }
   ],
   "source": [
    "xtreme_subsets = get_dataset_config_names(\"xtreme\")\n",
    "print(f\"XTREME has {len(xtreme_subsets)} configurations\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZvaDYaWt3pUS",
    "outputId": "72b11e4a-d125-4eb2-d6bc-de013d9a37a2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['PAN-X.af', 'PAN-X.ar', 'PAN-X.bg']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#  Let’s narrow the search by just looking for the configurations that start with “PAN”:\n",
    "\n",
    "panx_subsets = [s for s in xtreme_subsets if s.startswith(\"PAN\")]\n",
    "panx_subsets[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 329,
     "referenced_widgets": [
      "d7e613964c8b4103b4b718df31da6f7b",
      "11f7b37d0b4d4939a7a1ecec4a411bb1",
      "7222b506812340e3ba3bf8f9d76042d0",
      "80e3d8e2a3bb47ddbf76225dfdeb03cc",
      "caf40dc96d5e460884a2aa71da7b1b0b",
      "de7d70c071094a678fe89ba73fcdf442",
      "56baee0a1ce94b2c820c271adf32b801",
      "4d5747d0052b48dfab6794ebdb1930b0",
      "be8698ef584c4c7cb7a99cd7e4c8d959",
      "4524099fda054038a5bef33ca175dd14",
      "b595a25e58b04a459e6ea3ddeffba9dc"
     ]
    },
    "id": "GGp4QMjk3pUU",
    "outputId": "7ab12b80-cb05-4b11-b109-e19a242b19f4"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reusing dataset xtreme (/root/.cache/huggingface/datasets/xtreme/PAN-X.de/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7e613964c8b4103b4b718df31da6f7b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 20000\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 10000\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 10000\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "load_dataset(\"xtreme\", name=\"PAN-X.de\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "AV6iepCL3pUW"
   },
   "outputs": [],
   "source": [
    "languages = [\"de\", \"fr\", \"it\", \"en\"]\n",
    "percentage_lang_spoken = [0.629, 0.229, 0.084, 0.059]\n",
    "# Return a DatasetDict if a key doesn't exist\n",
    "panx_ds_combined = defaultdict(DatasetDict)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 442,
     "referenced_widgets": [
      "ca2f707159c543048e3eac987fff4578",
      "6a076dbd5f8b4009b4b8be6b00adb37c",
      "f86eb20bed6540efb3f931dfa142d7bc",
      "49bc2c3640174600b8f7023d5d97db6c",
      "9e6f411b91be4631970f9f92116f29e8",
      "f5837d6a41224c2ea7c165d4e6cd0f49",
      "a89869b1ca3647f1995eb084a55d822b",
      "bb8e24c80a1a4b38bf11f2b4ee00e92f",
      "6ccf7b324e3c400eaa454c0c11a3d04c",
      "ccc243f8298e40b78cb90f56f4ab5f3e",
      "24977036948944e3960e0f7654bb9150",
      "c5247e0ac1164f479eac9caebdd8f830",
      "33e629ed2e674752a91b2bfd4872992e",
      "4f448e3ebd8741548983425ed380ed14",
      "04d740ecc336428abf0cf4a790b9ffbd",
      "727ee0ced86b4cc996256a513e7961de",
      "154bdbef9b884e4084667340291dc3d3",
      "2e746d5c0053487bab87f2d02b164d9f",
      "81bf416562244225aa988dbe0b3b1bc5",
      "20d8e5de43b54c2690161d869571444d",
      "4a2e03065d1d4ee28f85385299827c0a",
      "600b1c7ca7864e419edfd33e6a702ab9",
      "9b3fca254ee24be997161ff32ed41910",
      "0594d6b5cecc4b01bdbefdb36a329995",
      "7294bc8dcff24989a8ec684eb095063a",
      "ae54f863ffee49a0a29a0133f36a3e71",
      "815b038be78b446988e4a3af396e11d1",
      "06101367c25445098881422afb123580",
      "7cad496137b04ef2acd1c84f5162a0a8",
      "787a3ead23584012a93a053fdf678bb2",
      "1e7c1b4e2a184835a55a9f6926399d2a",
      "b57cef29cc2948f8803504790aed8a16",
      "f5045a19461e4740a5a0848561753c14",
      "82e317f3fb1045adbb37bf162ce0e510",
      "785311ad23c14f2dbed096a9b054813f",
      "b9f7a0863a3045d39c328bc144bec0f8",
      "cd1d3d5dd7a54b20be5130e60ce595e7",
      "ca989bbad49c4865b2b8b8987a139c42",
      "4b0e2338976d4ababfb13410b1a77a43",
      "580c3d39bc774beda69eace3e4e00cdf",
      "364ac5a6b4d146eb8239c608e8c76fd2",
      "49bf43c1b30c4dbda112572d88d3ae89",
      "7854caba960e425e9518a172af7f96a7",
      "11ec68d5eb0544c797fd95b73555d57e"
     ]
    },
    "id": "1AdOff0c3pUX",
    "outputId": "5724d807-d4e1-464d-f5d2-6279024a9f62"
   },
   "outputs": [],
   "source": [
    "\n",
    "for lang, percentage in zip(languages, percentage_lang_spoken):\n",
    "    # Load monolingual corpus\n",
    "    ds = load_dataset(\"xtreme\", name=f\"PAN-X.{lang}\")\n",
    "    # Shuffle and downsample each train_test_val according to spoken proportion\n",
    "    for train_test_val in ds:\n",
    "        panx_ds_combined[lang][train_test_val] = (\n",
    "            ds[train_test_val]\n",
    "            .shuffle(seed=0)\n",
    "            .select(range(int(percentage * ds[train_test_val].num_rows))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OT-tALSp3pUX",
    "outputId": "42552020-9bac-488b-ef51-5d97b3286463"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(datasets.dataset_dict.DatasetDict,\n",
       "            {'de': DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 12580\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 6290\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 6290\n",
       "    })\n",
       "}),\n",
       "             'en': DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 1180\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 590\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 590\n",
       "    })\n",
       "}),\n",
       "             'fr': DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 4580\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 2290\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 2290\n",
       "    })\n",
       "}),\n",
       "             'it': DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 1680\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 840\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 840\n",
       "    })\n",
       "})})"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "panx_ds_combined"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81
    },
    "id": "0Hhf6m1P3pUZ",
    "outputId": "06d948af-a009-4828-9f4a-16f8913e50df"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-eefaa9b1-5b5b-4520-9ab4-3e9404c7a1ad\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>de</th>\n",
       "      <th>fr</th>\n",
       "      <th>it</th>\n",
       "      <th>en</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Number of training examples</th>\n",
       "      <td>12580</td>\n",
       "      <td>4580</td>\n",
       "      <td>1680</td>\n",
       "      <td>1180</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-eefaa9b1-5b5b-4520-9ab4-3e9404c7a1ad')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-eefaa9b1-5b5b-4520-9ab4-3e9404c7a1ad button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-eefaa9b1-5b5b-4520-9ab4-3e9404c7a1ad');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                                de    fr    it    en\n",
       "Number of training examples  12580  4580  1680  1180"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "pd.DataFrame({lang: [panx_ds_combined[lang][\"train\"].num_rows] for lang in languages},\n",
    "             index=[\"Number of training examples\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5DOdSi2c3pUa",
    "outputId": "09923f65-9f25-4f9b-99f2-a4ba0c787e5d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokens: ['2.000', 'Einwohnern', 'an', 'der', 'Danziger', 'Bucht', 'in', 'der', 'polnischen', 'Woiwodschaft', 'Pommern', '.']\n",
      "ner_tags: [0, 0, 0, 0, 5, 6, 0, 0, 5, 5, 6, 0]\n",
      "langs: ['de', 'de', 'de', 'de', 'de', 'de', 'de', 'de', 'de', 'de', 'de', 'de']\n"
     ]
    }
   ],
   "source": [
    "element = panx_ds_combined[\"de\"][\"train\"][0]\n",
    "\n",
    "for key, value in element.items():\n",
    "    print(f\"{key}: {value}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The keys here correspond to the column names of an Arrow table, while the values denote the entries in each column. \n",
    "\n",
    "In particular, we see that the ner_tags column corresponds to the mapping of each entity to a class ID. Each number in this class ID for ner_tags represents an entity. We can Convert the number to a label name for more information:\n",
    "\n",
    "https://huggingface.co/docs/transformers/tasks/token_classification\n",
    "\n",
    "This class IDs are not understandable to the human eye, so let’s create a new column with the familiar LOC, PER, and ORG tags. \n",
    "\n",
    "To do this, the first thing to notice is that our **Dataset object has a features attribute that specifies the underlying data types associated with each column**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ouixLbWJ3pUa",
    "outputId": "a332be17-ccab-42fe-a300-d49423570746"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['tokens', 'ner_tags', 'langs'],\n",
       "    num_rows: 12580\n",
       "})"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "panx_ds_combined[\"de\"]['train']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cGeOz0fQ3pUb",
    "outputId": "34e17a24-f015-4422-e263-f5b48db7835d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokens: Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)\n",
      "ner_tags: Sequence(feature=ClassLabel(num_classes=7, names=['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC'], id=None), length=-1, id=None)\n",
      "langs: Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)\n"
     ]
    }
   ],
   "source": [
    "for key, value in panx_ds_combined[\"de\"][\"train\"].features.items():\n",
    "    print(f\"{key}: {value}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HH8pZeSQ3pUb",
    "outputId": "b205b386-0ab6-4a40-d481-1e64367ee781"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ClassLabel(num_classes=7, names=['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC'], id=None)\n"
     ]
    }
   ],
   "source": [
    "tags = panx_ds_combined[\"de\"][\"train\"].features[\"ner_tags\"].feature\n",
    "print(tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HlsLz_Xp3pUc",
    "outputId": "69da6286-70bd-44ba-fe68-fe8e532f2761"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 12580\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 6290\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs'],\n",
       "        num_rows: 6290\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "panx_ds_combined[\"de\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XW0Lwb0d3pUc",
    "outputId": "e9fb4c45-2b0c-4167-bbda-f33e45bcd451"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.de/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-204dccd4212a347d.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.de/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-28cc9172fa0bd1d3.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.de/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-973a02444af226e2.arrow\n"
     ]
    }
   ],
   "source": [
    "\n",
    "def create_tag_names(batch):\n",
    "    return {\"ner_tags_str\": [tags.int2str(idx) for idx in batch[\"ner_tags\"]]}\n",
    "\n",
    "panx_de = panx_ds_combined[\"de\"].map(create_tag_names)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ClassLabel.int2str() method converts integer => class name string. In here, it will create a new column named **'ner_tags_str'** in our training set with class names for each tag. \n",
    "\n",
    "https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/main_classes#datasets.ClassLabel.int2str\n",
    "\n",
    "The map() method returns a dict with the key corresponding to the new column name and the value as a list of class names:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "90ffvzQK3pUc",
    "outputId": "5d285b9e-f64f-41f9-d80f-a3c2a0e3b4a7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs', 'ner_tags_str'],\n",
       "        num_rows: 12580\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs', 'ner_tags_str'],\n",
       "        num_rows: 6290\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['tokens', 'ner_tags', 'langs', 'ner_tags_str'],\n",
       "        num_rows: 6290\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "panx_de"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WhMl47t53pUd",
    "outputId": "37e07de5-1406-4d8b-e5c6-938982d2584c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'langs': ['de',\n",
       "  'de',\n",
       "  'de',\n",
       "  'de',\n",
       "  'de',\n",
       "  'de',\n",
       "  'de',\n",
       "  'de',\n",
       "  'de',\n",
       "  'de',\n",
       "  'de',\n",
       "  'de'],\n",
       " 'ner_tags': [0, 0, 0, 0, 5, 6, 0, 0, 5, 5, 6, 0],\n",
       " 'ner_tags_str': ['O',\n",
       "  'O',\n",
       "  'O',\n",
       "  'O',\n",
       "  'B-LOC',\n",
       "  'I-LOC',\n",
       "  'O',\n",
       "  'O',\n",
       "  'B-LOC',\n",
       "  'B-LOC',\n",
       "  'I-LOC',\n",
       "  'O'],\n",
       " 'tokens': ['2.000',\n",
       "  'Einwohnern',\n",
       "  'an',\n",
       "  'der',\n",
       "  'Danziger',\n",
       "  'Bucht',\n",
       "  'in',\n",
       "  'der',\n",
       "  'polnischen',\n",
       "  'Woiwodschaft',\n",
       "  'Pommern',\n",
       "  '.']}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " panx_de[\"train\"][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 112
    },
    "id": "hWB1fn9b3pUd",
    "outputId": "94583084-35e5-47f7-cab9-3311c9576dae"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-21df8efd-5fd2-41e4-aaa9-45d2b5e3fada\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Tokens</th>\n",
       "      <td>2.000</td>\n",
       "      <td>Einwohnern</td>\n",
       "      <td>an</td>\n",
       "      <td>der</td>\n",
       "      <td>Danziger</td>\n",
       "      <td>Bucht</td>\n",
       "      <td>in</td>\n",
       "      <td>der</td>\n",
       "      <td>polnischen</td>\n",
       "      <td>Woiwodschaft</td>\n",
       "      <td>Pommern</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tags</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-21df8efd-5fd2-41e4-aaa9-45d2b5e3fada')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-21df8efd-5fd2-41e4-aaa9-45d2b5e3fada button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-21df8efd-5fd2-41e4-aaa9-45d2b5e3fada');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "           0           1   2    3         4      5   6    7           8   \\\n",
       "Tokens  2.000  Einwohnern  an  der  Danziger  Bucht  in  der  polnischen   \n",
       "Tags        O           O   O    O     B-LOC  I-LOC   O    O       B-LOC   \n",
       "\n",
       "                  9        10 11  \n",
       "Tokens  Woiwodschaft  Pommern  .  \n",
       "Tags           B-LOC    I-LOC  O  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "de_example = panx_de[\"train\"][0]\n",
    "pd.DataFrame([de_example[\"tokens\"], de_example[\"ner_tags_str\"]],\n",
    "['Tokens', 'Tags'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 143
    },
    "id": "tOnSAEQz3pUd",
    "outputId": "6e51ba2c-b01c-4756-c83a-e8d52c4e0628"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-da599ca8-155b-4ba7-a96f-9ebbba9ff48c\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LOC</th>\n",
       "      <th>ORG</th>\n",
       "      <th>PER</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>train</th>\n",
       "      <td>6186</td>\n",
       "      <td>5366</td>\n",
       "      <td>5810</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>validation</th>\n",
       "      <td>3172</td>\n",
       "      <td>2683</td>\n",
       "      <td>2893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test</th>\n",
       "      <td>3180</td>\n",
       "      <td>2573</td>\n",
       "      <td>3071</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-da599ca8-155b-4ba7-a96f-9ebbba9ff48c')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-da599ca8-155b-4ba7-a96f-9ebbba9ff48c button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-da599ca8-155b-4ba7-a96f-9ebbba9ff48c');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "             LOC   ORG   PER\n",
       "train       6186  5366  5810\n",
       "validation  3172  2683  2893\n",
       "test        3180  2573  3071"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "def get_ner_frequencies(panx_de):\n",
    "  split2freqs = defaultdict(Counter)\n",
    "  for split, dataset in panx_de.items():\n",
    "      for row in dataset[\"ner_tags_str\"]:\n",
    "          for tag in row:\n",
    "              if tag.startswith(\"B\"):\n",
    "                  tag_type = tag.split(\"-\")[1]\n",
    "                  split2freqs[split][tag_type] += 1\n",
    "  return split2freqs\n",
    "\n",
    "freqs = get_ner_frequencies(panx_de)\n",
    "\n",
    "pd.DataFrame.from_dict(freqs, orient=\"index\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a Custom Model for Token Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "QdcsDQaH3pUf"
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "from matplotlib import pyplot as plt\n",
    "from transformers import XLMRobertaConfig\n",
    "from transformers.modeling_outputs import TokenClassifierOutput\n",
    "from transformers.models.roberta.modeling_roberta import RobertaModel\n",
    "from transformers.models.roberta.modeling_roberta import RobertaPreTrainedModel\n",
    "\n",
    "class XLMRobertaForTokenClassification(RobertaPreTrainedModel):\n",
    "    \"\"\"\n",
    "    This class extends the RobertaPreTrainedModel class for token classification tasks.\n",
    "    The model architecture is based on the XLMRoberta model.\n",
    "    \"\"\"\n",
    "    config_class = XLMRobertaConfig\n",
    "\n",
    "    def __init__(self, config):\n",
    "        \"\"\"\n",
    "        Constructor for the XLMRobertaForTokenClassification class.\n",
    "        \n",
    "        Parameters:\n",
    "        config (XLMRobertaConfig): Configuration object containing information about how to build the model.\n",
    "        \"\"\"\n",
    "        super().__init__(config)\n",
    "        self.num_labels = config.num_labels\n",
    "        # Load model body\n",
    "        self.roberta = RobertaModel(config, add_pooling_layer=False)\n",
    "        # Set up token classification head\n",
    "        self.dropout = nn.Dropout(config.hidden_dropout_prob)\n",
    "        self.classifier = nn.Linear(config.hidden_size, config.num_labels)\n",
    "        # Load and initialize weights\n",
    "        self.init_weights()\n",
    "\n",
    "    def forward(self, input_ids=None, attention_mask=None, token_type_ids=None, \n",
    "                labels=None, **kwargs):\n",
    "        \"\"\"\n",
    "        Forward method for the XLMRobertaForTokenClassification class.\n",
    "        \n",
    "        Parameters:\n",
    "        input_ids (torch.Tensor): Tensor of input ids of shape (batch_size, sequence_length).\n",
    "        attention_mask (torch.Tensor): Tensor of attention masks of shape (batch_size, sequence_length).\n",
    "        token_type_ids (torch.Tensor): Tensor of token type ids of shape (batch_size, sequence_length).\n",
    "        labels (torch.Tensor): Tensor of labels of shape (batch_size, sequence_length).\n",
    "        \n",
    "        Returns:\n",
    "        TokenClassifierOutput: An object that contains the loss and the logits.\n",
    "        \"\"\"\n",
    "        # Use model body to get encoder representations\n",
    "        outputs = self.roberta(input_ids, attention_mask=attention_mask,\n",
    "                               token_type_ids=token_type_ids, **kwargs)\n",
    "        # Apply classifier to encoder representation\n",
    "        sequence_output = self.dropout(outputs[0])\n",
    "        logits = self.classifier(sequence_output)\n",
    "        # Calculate losses\n",
    "        loss = None\n",
    "        if labels is not None:\n",
    "            loss_fct = nn.CrossEntropyLoss()\n",
    "            loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))\n",
    "        # Return model output object\n",
    "        return TokenClassifierOutput(loss=loss, logits=logits, \n",
    "                                     hidden_states=outputs.hidden_states, \n",
    "                                     attentions=outputs.attentions)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------------\n",
    "## TFTokenClassifierOutput\n",
    "\n",
    "\n",
    "https://huggingface.co/docs/transformers/main_classes/output#transformers.modeling_tf_outputs.TFTokenClassifierOutput\n",
    "\n",
    "\n",
    "All models have outputs that are instances of subclasses of ModelOutput. Those are data structures containing all the information returned by the model, but that can also be used as tuples or dictionaries.\n",
    "\n",
    "\n",
    "The outputs object is a SequenceClassifierOutput, as we can see in the documentation of that class below, it means it has an optional loss, a logits an optional hidden_states and an optional attentions attribute. Here we have the loss since we passed along labels, but we don’t have hidden_states and attentions because we didn’t pass output_hidden_states=True or output_attentions=True.\n",
    "\n",
    "\n",
    "-------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "NDlNptTz3pUf"
   },
   "outputs": [],
   "source": [
    "index2tag = {idx: tag for idx, tag in enumerate(tags.names)}\n",
    "tag2index = {tag: idx for idx, tag in enumerate(tags.names)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "ZrE2Sqmc3pUf",
    "outputId": "f9688860-eb62-4c6e-9397-c314e020628f"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "\"\\n{0: 'O',\\n 1: 'B-PER',\\n 2: 'I-PER',\\n 3: 'B-ORG',\\n 4: 'I-ORG',\\n 5: 'B-LOC',\\n 6: 'I-LOC'} \""
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index2tag\n",
    "'''\n",
    "{0: 'O',\n",
    " 1: 'B-PER',\n",
    " 2: 'I-PER',\n",
    " 3: 'B-ORG',\n",
    " 4: 'I-ORG',\n",
    " 5: 'B-LOC',\n",
    " 6: 'I-LOC'} '''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "eKzjGaWv3pUg",
    "outputId": "b48c60a5-d551-4beb-a41a-0d82b948cb38"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'B-LOC': 5,\n",
       " 'B-ORG': 3,\n",
       " 'B-PER': 1,\n",
       " 'I-LOC': 6,\n",
       " 'I-ORG': 4,\n",
       " 'I-PER': 2,\n",
       " 'O': 0}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tag2index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "9m6OjHGW3pUg"
   },
   "outputs": [],
   "source": [
    "bert_model_name = \"bert-base-cased\"\n",
    "xlmr_model_name = \"xlm-roberta-base\"\n",
    "\n",
    "bert_tokenizer = AutoTokenizer.from_pretrained(bert_model_name)\n",
    "xlmr_tokenizer = AutoTokenizer.from_pretrained(xlmr_model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "E6ArQ_C_3pUg"
   },
   "outputs": [],
   "source": [
    "\n",
    "xlmr_config = AutoConfig.from_pretrained(xlmr_model_name, \n",
    "                                         num_labels=tags.num_classes,\n",
    "                                         id2label=index2tag, label2id=tag2index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5f8ZklkN3pUh",
    "outputId": "c0c6668f-47a5-4df7-fe34-f1bace10cefd"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at xlm-roberta-base were not used when initializing XLMRobertaForTokenClassification: ['lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.weight', 'roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']\n",
      "- This IS expected if you are initializing XLMRobertaForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing XLMRobertaForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of XLMRobertaForTokenClassification were not initialized from the model checkpoint at xlm-roberta-base and are newly initialized: ['roberta.embeddings.position_ids', 'classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "# device = 'cpu'\n",
    "\n",
    "xlmr_my_model = (XLMRobertaForTokenClassification\n",
    "              .from_pretrained(xlmr_model_name, config=xlmr_config)\n",
    "              .to(device))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "mSOUfwKQ3pUh"
   },
   "outputs": [],
   "source": [
    "example_string = 'World will be changed with AI'\n",
    "\n",
    "bert_tokens = bert_tokenizer(example_string).tokens()\n",
    "xlmr_tokens = xlmr_tokenizer(example_string).tokens()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "d5QADQRG3pUh",
    "outputId": "86bab655-4530-40a2-cade-4178682287de"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<s>', '▁World', '▁will', '▁be', '▁changed', '▁with', '▁AI', '</s>']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xlmr_tokens\n",
    "# ['<s>', '▁World', '▁will', '▁be', '▁changed', '▁with', '▁AI', '</s>']['<s>', '▁World', '▁will', '▁be', '▁changed', '▁with', '▁AI', '</s>']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "NyYRya1R3pUi"
   },
   "outputs": [],
   "source": [
    "def tag_text(text, tags, model, tokenizer):\n",
    "    \"\"\"\n",
    "    Function to tag text using a pretrained model and tokenizer.\n",
    "\n",
    "    Parameters:\n",
    "    text (str): Text to be tagged.\n",
    "    tags (Tags): An instance of the Tags class that maps class indices to names.\n",
    "    model (Model): Pretrained model for named entity recognition.\n",
    "    tokenizer (Tokenizer): Pretrained tokenizer corresponding to the model.\n",
    "\n",
    "    Returns:\n",
    "    pd.DataFrame: DataFrame containing the original tokens and their corresponding tags.\n",
    "    \"\"\"\n",
    "    # Tokenize the text, preserving special characters\n",
    "    tokens = tokenizer(text).tokens()\n",
    "\n",
    "    # Encode the tokens into their corresponding IDs\n",
    "    # 'return_tensors=\"pt\"' indicates that we want PyTorch tensors\n",
    "    # '.to(device)' sends the tensors to the GPU if one is available\n",
    "    input_ids = tokenizer(text, return_tensors=\"pt\").input_ids.to(device)\n",
    "\n",
    "    # Print the input IDs for debugging purposes\n",
    "    print('input_ids ', input_ids)\n",
    "\n",
    "    # Get the model's predictions as a distribution over possible classes\n",
    "    # Each token is associated with a probability distribution over the 7 possible NER tags\n",
    "    outputs = model(input_ids)[0]\n",
    "    print(f\"Shape of outputs: {outputs.shape}\")\n",
    "\n",
    "    # Take the argmax over the tag dimension to get the most likely class for each token\n",
    "    predictions = torch.argmax(outputs, dim=2)\n",
    "    print('predictions ', predictions)\n",
    "\n",
    "    # Convert the predictions from class indices to class names using the 'tags' object\n",
    "    # '.cpu().numpy()' moves the tensor from GPU to CPU and converts it to a NumPy array\n",
    "    preds = [tags.names[p] for p in predictions[0].cpu().numpy()]\n",
    "\n",
    "    # Return the tokens and their corresponding tags as a DataFrame\n",
    "    return pd.DataFrame([tokens, preds], index=[\"Tokens\", \"Tags\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "i4jBT6Ta3pUi",
    "outputId": "77baa889-e936-401f-8b59-a13de2efad45"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ClassLabel(num_classes=7, names=['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC'], id=None)\n",
      "PreTrainedTokenizerFast(name_or_path='xlm-roberta-base', vocab_size=250002, model_max_len=512, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'sep_token': '</s>', 'pad_token': '<pad>', 'cls_token': '<s>', 'mask_token': AddedToken(\"<mask>\", rstrip=False, lstrip=True, single_word=False, normalized=False)})\n"
     ]
    }
   ],
   "source": [
    "print(tags)\n",
    "print(xlmr_tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 181
    },
    "id": "5pw_TSyc3pUi",
    "outputId": "2b9889e8-b636-43c4-eb90-4ead5728db84"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_ids  tensor([[    0,  6661,  1221,   186, 98816,   678, 38730,     2]],\n",
      "       device='cuda:0')\n",
      "Shape of outputs: torch.Size([1, 8, 7])\n",
      "predictions  tensor([[2, 2, 2, 2, 2, 2, 2, 2]], device='cuda:0')\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-498c23f5-ea56-4b90-a0b9-26d0ac85eaee\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Tokens</th>\n",
       "      <td>&lt;s&gt;</td>\n",
       "      <td>▁World</td>\n",
       "      <td>▁will</td>\n",
       "      <td>▁be</td>\n",
       "      <td>▁changed</td>\n",
       "      <td>▁with</td>\n",
       "      <td>▁AI</td>\n",
       "      <td>&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tags</th>\n",
       "      <td>I-PER</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>I-PER</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-498c23f5-ea56-4b90-a0b9-26d0ac85eaee')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-498c23f5-ea56-4b90-a0b9-26d0ac85eaee button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-498c23f5-ea56-4b90-a0b9-26d0ac85eaee');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "            0       1      2      3         4      5      6      7\n",
       "Tokens    <s>  ▁World  ▁will    ▁be  ▁changed  ▁with    ▁AI   </s>\n",
       "Tags    I-PER   I-PER  I-PER  I-PER     I-PER  I-PER  I-PER  I-PER"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tag_text(example_string, tags, xlmr_my_model, xlmr_tokenizer )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "xzcNm2jR3pUj"
   },
   "outputs": [],
   "source": [
    "words, labels = de_example[\"tokens\"], de_example[\"ner_tags\"]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we tokenize each word and use the **`is_split_into_words`** argument to tell\n",
    "the tokenizer that our input sequence has already been split into words:\n",
    "\n",
    "**From Doc - is_split_into_words (bool, optional, defaults to False)** — If set to True, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. This is useful for NER or token classification.\n",
    "\n",
    "\n",
    "https://github.com/huggingface/transformers/issues/8217\n",
    "\n",
    "#### **`is_split_into_words`** DOES NOT MEAN that the text was already pre-tokenized. It just means that the string was split into words (not tokens), i.e., split on spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "id": "nPHOIZLb3pUj"
   },
   "outputs": [],
   "source": [
    "tokenized_input = xlmr_tokenizer(de_example[\"tokens\"], is_split_into_words=True)\n",
    "\n",
    "tokens = xlmr_tokenizer.convert_ids_to_tokens(tokenized_input[\"input_ids\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 110
    },
    "id": "EoAyaFbJ3pUj",
    "outputId": "4e3c68da-90e3-4002-b394-300fa75d9f5e"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-a0595283-cd1a-450b-b768-90fc4093377c\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Tokens</th>\n",
       "      <td>&lt;s&gt;</td>\n",
       "      <td>▁2.000</td>\n",
       "      <td>▁Einwohner</td>\n",
       "      <td>n</td>\n",
       "      <td>▁an</td>\n",
       "      <td>▁der</td>\n",
       "      <td>▁Dan</td>\n",
       "      <td>zi</td>\n",
       "      <td>ger</td>\n",
       "      <td>▁Buch</td>\n",
       "      <td>...</td>\n",
       "      <td>▁Wo</td>\n",
       "      <td>i</td>\n",
       "      <td>wod</td>\n",
       "      <td>schaft</td>\n",
       "      <td>▁Po</td>\n",
       "      <td>mmer</td>\n",
       "      <td>n</td>\n",
       "      <td>▁</td>\n",
       "      <td>.</td>\n",
       "      <td>&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 25 columns</p>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a0595283-cd1a-450b-b768-90fc4093377c')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-a0595283-cd1a-450b-b768-90fc4093377c button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-a0595283-cd1a-450b-b768-90fc4093377c');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "         0       1           2  3    4     5     6   7    8      9   ...   15  \\\n",
       "Tokens  <s>  ▁2.000  ▁Einwohner  n  ▁an  ▁der  ▁Dan  zi  ger  ▁Buch  ...  ▁Wo   \n",
       "\n",
       "       16   17      18   19    20 21 22 23    24  \n",
       "Tokens  i  wod  schaft  ▁Po  mmer  n  ▁  .  </s>  \n",
       "\n",
       "[1 rows x 25 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "pd.DataFrame([tokens], index=[\"Tokens\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example we can see that the tokenizer has split “Einwohnern” into two\n",
    "subwords, “▁Einwohner” and “n”. \n",
    "\n",
    "Since we’re following the convention that only “▁Einwohner” should be associated with the B-LOC label, we need a way to mask the subword representations after the first subword. \n",
    "\n",
    "And for this purpose, tokenizer contains a word_ids() function that can help us achieve this:\n",
    "\n",
    "\n",
    "## word_ids()\n",
    "\n",
    "https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.BatchEncoding.word_ids  \n",
    "\n",
    "Return a list mapping the tokens to their actual word in the initial sentence for a fast tokenizer. It Returns a list indicating the word corresponding to each token. \n",
    "\n",
    "### In word_ids() the Special tokens added by the tokenizer are mapped to None and other tokens are mapped to the index of their corresponding word (several tokens will be mapped to the same word index if they are parts of that word).\n",
    "\n",
    "Returns a list like below..\n",
    "\n",
    "```\n",
    "[None, 0, 1, 1, 2, 3, 4, 4, 4, 5, 5, 6, 7, 8, 8, 9, 9, 9, 9, 10, 10, 10, 11, 11, None]\n",
    "0\t1\t2\t3\t4\t5\t6\t7\t8\t9\t...\t15\t16\t17\t18\t19\t20\t21\t22\t23\t24\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 159
    },
    "id": "uCiINLnM3pUk",
    "outputId": "68b7add6-35b4-4451-804c-04eddc5919ba"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "word_ids  [None, 0, 1, 1, 2, 3, 4, 4, 4, 5, 5, 6, 7, 8, 8, 9, 9, 9, 9, 10, 10, 10, 11, 11, None]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-cd955155-20b7-4098-b667-b89e53ee1ccf\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Tokens</th>\n",
       "      <td>&lt;s&gt;</td>\n",
       "      <td>▁2.000</td>\n",
       "      <td>▁Einwohner</td>\n",
       "      <td>n</td>\n",
       "      <td>▁an</td>\n",
       "      <td>▁der</td>\n",
       "      <td>▁Dan</td>\n",
       "      <td>zi</td>\n",
       "      <td>ger</td>\n",
       "      <td>▁Buch</td>\n",
       "      <td>...</td>\n",
       "      <td>▁Wo</td>\n",
       "      <td>i</td>\n",
       "      <td>wod</td>\n",
       "      <td>schaft</td>\n",
       "      <td>▁Po</td>\n",
       "      <td>mmer</td>\n",
       "      <td>n</td>\n",
       "      <td>▁</td>\n",
       "      <td>.</td>\n",
       "      <td>&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Word IDs</th>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 25 columns</p>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cd955155-20b7-4098-b667-b89e53ee1ccf')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-cd955155-20b7-4098-b667-b89e53ee1ccf button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-cd955155-20b7-4098-b667-b89e53ee1ccf');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "            0       1           2  3    4     5     6   7    8      9   ...  \\\n",
       "Tokens     <s>  ▁2.000  ▁Einwohner  n  ▁an  ▁der  ▁Dan  zi  ger  ▁Buch  ...   \n",
       "Word IDs  None       0           1  1    2     3     4   4    4      5  ...   \n",
       "\n",
       "           15 16   17      18   19    20  21  22  23    24  \n",
       "Tokens    ▁Wo  i  wod  schaft  ▁Po  mmer   n   ▁   .  </s>  \n",
       "Word IDs    9  9    9       9   10    10  10  11  11  None  \n",
       "\n",
       "[2 rows x 25 columns]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "word_ids = tokenized_input.word_ids()\n",
    "print('word_ids ', word_ids)\n",
    "\n",
    "pd.DataFrame([tokens, word_ids], index=[\"Tokens\", \"Word IDs\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "id": "oilJouN33pUk",
    "outputId": "5c47fd0c-1288-42e7-f22a-c2e97116f28d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "label_ids  [-100, 0, 0, -100, 0, 0, 5, -100, -100, 6, -100, 0, 0, 5, -100, 5, -100, -100, -100, 6, -100, -100, 0, -100, -100]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-a3cfd744-52ed-4fba-9916-70b1d60500bd\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Tokens</th>\n",
       "      <td>&lt;s&gt;</td>\n",
       "      <td>▁2.000</td>\n",
       "      <td>▁Einwohner</td>\n",
       "      <td>n</td>\n",
       "      <td>▁an</td>\n",
       "      <td>▁der</td>\n",
       "      <td>▁Dan</td>\n",
       "      <td>zi</td>\n",
       "      <td>ger</td>\n",
       "      <td>▁Buch</td>\n",
       "      <td>...</td>\n",
       "      <td>▁Wo</td>\n",
       "      <td>i</td>\n",
       "      <td>wod</td>\n",
       "      <td>schaft</td>\n",
       "      <td>▁Po</td>\n",
       "      <td>mmer</td>\n",
       "      <td>n</td>\n",
       "      <td>▁</td>\n",
       "      <td>.</td>\n",
       "      <td>&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Word IDs</th>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Label IDs</th>\n",
       "      <td>-100</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-100</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>-100</td>\n",
       "      <td>-100</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>-100</td>\n",
       "      <td>-100</td>\n",
       "      <td>-100</td>\n",
       "      <td>6</td>\n",
       "      <td>-100</td>\n",
       "      <td>-100</td>\n",
       "      <td>0</td>\n",
       "      <td>-100</td>\n",
       "      <td>-100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Labels</th>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>...</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 25 columns</p>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a3cfd744-52ed-4fba-9916-70b1d60500bd')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-a3cfd744-52ed-4fba-9916-70b1d60500bd button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-a3cfd744-52ed-4fba-9916-70b1d60500bd');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "             0       1           2     3    4     5      6     7     8   \\\n",
       "Tokens      <s>  ▁2.000  ▁Einwohner     n  ▁an  ▁der   ▁Dan    zi   ger   \n",
       "Word IDs   None       0           1     1    2     3      4     4     4   \n",
       "Label IDs  -100       0           0  -100    0     0      5  -100  -100   \n",
       "Labels      IGN       O           O   IGN    O     O  B-LOC   IGN   IGN   \n",
       "\n",
       "              9   ...     15    16    17      18     19    20    21  22    23  \\\n",
       "Tokens     ▁Buch  ...    ▁Wo     i   wod  schaft    ▁Po  mmer     n   ▁     .   \n",
       "Word IDs       5  ...      9     9     9       9     10    10    10  11    11   \n",
       "Label IDs      6  ...      5  -100  -100    -100      6  -100  -100   0  -100   \n",
       "Labels     I-LOC  ...  B-LOC   IGN   IGN     IGN  I-LOC   IGN   IGN   O   IGN   \n",
       "\n",
       "             24  \n",
       "Tokens     </s>  \n",
       "Word IDs   None  \n",
       "Label IDs  -100  \n",
       "Labels      IGN  \n",
       "\n",
       "[4 rows x 25 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "previous_word_idx = None\n",
    "label_ids = []\n",
    "\n",
    "for word_idx in word_ids:\n",
    "    if word_idx is None or word_idx == previous_word_idx:\n",
    "        label_ids.append(-100)\n",
    "    elif word_idx != previous_word_idx:\n",
    "        label_ids.append(labels[word_idx])\n",
    "    # And then update previous_word_idx\n",
    "    # to be current word\n",
    "    previous_word_idx = word_idx\n",
    "\n",
    "print('label_ids ', label_ids)\n",
    "    \n",
    "labels = [index2tag[l] if l != -100 else \"IGN\" for l in label_ids]\n",
    "index = [\"Tokens\", \"Word IDs\", \"Label IDs\", \"Labels\"]\n",
    "\n",
    "pd.DataFrame([tokens, word_ids, label_ids, labels], index=index)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let’s set –100 as the label for these special tokens and the subwords we wish to mask during training:\n",
    "Why did we choose –100 as the ID to mask subword representations? The reason is\n",
    "that in PyTorch the cross-entropy loss class torch.nn.CrossEntropyLoss has an\n",
    "attribute called ignore_index whose value is –100. This index is ignored during\n",
    "training, so we can use it to ignore the tokens associated with consecutive\n",
    "subwords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "id": "gMgevxKU3pUk"
   },
   "outputs": [],
   "source": [
    "def tokenize_mask_modify_labels(examples):\n",
    "    \"\"\"\n",
    "    Function to tokenize the input text, mask certain tokens, and adjust the labels accordingly.\n",
    "\n",
    "    Parameters:\n",
    "    examples (dict): A dictionary containing the input text and labels. The input text (\"tokens\") is a list of words, \n",
    "                     and the labels (\"ner_tags\") are a corresponding list of NER tags.\n",
    "\n",
    "    Returns:\n",
    "    tokenized_inputs (dict): A dictionary containing the tokenized input text and modified labels. The tokenized input \n",
    "                             text is a list of token IDs, and the modified labels are a corresponding list of NER tags, \n",
    "                             with -100 indicating masked tokens.\n",
    "    \"\"\"\n",
    "    tokenized_inputs = xlmr_tokenizer(examples[\"tokens\"], truncation=True, \n",
    "                                      is_split_into_words=True)\n",
    "    labels = []\n",
    "    for idx, label in enumerate(examples[\"ner_tags\"]):\n",
    "        word_ids = tokenized_inputs.word_ids(batch_index=idx)\n",
    "        # word_ids() => Return a list mapping the tokens\n",
    "        # to their actual word in the initial sentence.\n",
    "        # It Returns a list indicating the word corresponding to each token. \n",
    "        previous_word_idx = None\n",
    "        label_ids = []\n",
    "        # Special tokens like `<s>` and `<\\s>` are mapped to None. \n",
    "        # Set –100 as the label for these special tokens and\n",
    "        # the subwords we wish to mask during training:\n",
    "        for word_idx in word_ids:\n",
    "            if word_idx is None or word_idx == previous_word_idx:\n",
    "                label_ids.append(-100)\n",
    "            else:\n",
    "                label_ids.append(label[word_idx])\n",
    "            previous_word_idx = word_idx\n",
    "        labels.append(label_ids)\n",
    "    tokenized_inputs[\"labels\"] = labels\n",
    "    return tokenized_inputs"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Map\n",
    "\n",
    "https://huggingface.co/docs/datasets/process#map\n",
    "\n",
    "Some of the more powerful applications of 🤗 Datasets come from using the map() function. The primary purpose of map() is to speed up processing functions. It allows you to apply a processing function to each example in a dataset, independently or in batches. This function can even create new rows and columns.\n",
    "\n",
    "**batched (bool, default False)** — Provide batch of examples to function.\n",
    "\n",
    "**remove_columns (Optional[Union[str, List[str]]], default None)** — Remove a selection of columns while doing the mapping. Columns will be removed before updating the examples with the output of function, i.e. if function is adding columns with names in remove_columns, these columns will be kept."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "id": "9kqh2cGs3pUl"
   },
   "outputs": [],
   "source": [
    "def encode_panx_dataset(corpus):\n",
    "    return corpus.map(tokenize_mask_modify_labels, batched=True, \n",
    "                      remove_columns=['langs', 'ner_tags', 'tokens'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 104,
     "referenced_widgets": [
      "1e9ccb7fc6cb40f8998e53666e5f720b",
      "6c558c28973d4d659fc9a4c7b01c9279",
      "f9ce323e0969454cb4facbf6f1246c2f",
      "f33d5356927141fabd5b6f2296eb3c15",
      "e314843794f6448492bc6248131bb54b",
      "f0fd1897d30d467b999076fb6da052ee",
      "b82f3962015b4b479398f7869417a198",
      "21dccd4f0cd54ba180fd6e5cccf4464a",
      "e1352047b2144c76aa4a7619677128e7",
      "c5c5d60cb9794a68ae13371e6f76a1da",
      "eb07821864444733b71a1e21c5f167a6"
     ]
    },
    "id": "F6uas5fd3pUl",
    "outputId": "6f371e1c-7160-4914-cb7d-607b8788132a"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.de/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-08d84fb9fdc079b2.arrow\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1e9ccb7fc6cb40f8998e53666e5f720b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/7 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.de/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-c1c2d298fc80b661.arrow\n"
     ]
    }
   ],
   "source": [
    "\n",
    "panx_de_encoded = encode_panx_dataset(panx_ds_combined[\"de\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "hLb4hvB93pUm",
    "outputId": "673700aa-0b40-433e-c606-82006d278cbf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "        MISC       0.00      0.00      0.00         1\n",
      "         PER       1.00      1.00      1.00         1\n",
      "\n",
      "   micro avg       0.50      0.50      0.50         2\n",
      "   macro avg       0.50      0.50      0.50         2\n",
      "weighted avg       0.50      0.50      0.50         2\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from seqeval.metrics import classification_report\n",
    "\n",
    "y_true = [[\"O\", \"O\", \"O\", \"B-MISC\", \"I-MISC\", \"I-MISC\", \"O\"],\n",
    "          [\"B-PER\", \"I-PER\", \"O\"]]\n",
    "y_pred = [[\"O\", \"O\", \"B-MISC\", \"I-MISC\", \"I-MISC\", \"I-MISC\", \"O\"],\n",
    "          [\"B-PER\", \"I-PER\", \"O\"]]\n",
    "print(classification_report(y_true, y_pred))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The seqeval package expects the predictions and labels as lists of lists, with\n",
    "each list corresponding to a single example in our validation or test sets. To\n",
    "integrate these metrics during training, we need a function that can take the\n",
    "outputs of the model and convert them into the lists that seqeval expects. The\n",
    "following does the trick by ensuring we ignore the label IDs associated with\n",
    "subsequent subwords:\n",
    "\n",
    "\n",
    "-----\n",
    "\n",
    "Noting down the below variables, as was calculated earlier in this NB, but see that in the following function `generate_list_for_compute_metrics` these same variables are not used.\n",
    "\n",
    "label_ids => [-100, 'IGN', 'O', -100, 'O', 'IGN', 'O', -100, -100, 'O', -100, 'B-LOC', 'IGN', 'IGN', -100, 'I-LOC', -100, -100, -100, 'IGN', -100, -100, 'O', -100, -100]\n",
    "\n",
    "\n",
    "predictions will be =>  tensor([[2, 2, 2, 2, 2, 2, 2, 2]])\n",
    "\n",
    "\n",
    "-----\n",
    "\n",
    "## EvalPrediction\n",
    "\n",
    "#### EvalPrediction - Inside `compute_metrics()` when I print `eval_pred` - It Will output below\n",
    "\n",
    "<transformers.trainer_utils.EvalPrediction>\n",
    "\n",
    "https://huggingface.co/docs/transformers/internal/trainer_utils#transformers.EvalPrediction\n",
    "\n",
    "\n",
    "### `class transformers.EvalPrediction`\n",
    "\n",
    "Parameters\n",
    "\n",
    "* predictions (np.ndarray) — Predictions of the model.\n",
    "* label_ids (np.ndarray) — Targets to be matched.\n",
    "* inputs (np.ndarray, optional) —\n",
    "\n",
    "Evaluation output (always contains labels), to be used to compute metrics.\n",
    "\n",
    "### What does EvalPrediction.predictions contain exactly?\n",
    "\n",
    "The Trainer will put in **`predictions`** everything your model returns (apart from the loss).\n",
    "\n",
    "### When I print `eval_pred.predictions` - I get below a 3D-Tensor\n",
    "\n",
    "```\n",
    "[[[ 5.5185753e-01 -2.7944833e-01 -7.5236917e-01 ...  3.5712823e-01\n",
    "   -6.6847444e-01  2.7783272e-01]\n",
    "  [-6.0417420e-01  1.7045190e+00 -3.0190976e+00 ... -8.1113666e-01\n",
    "    1.5366187e+00 -3.7904561e+00]\n",
    "  [-5.0854206e-01 -1.9779130e+00  9.6103489e-02 ...  4.6591806e+00\n",
    "   -1.7985896e+00 -6.7350179e-01]\n",
    "```\n",
    "\n",
    "And note within the `generate_list_for_compute_metrics` function I do below to get the Preds\n",
    "\n",
    "`preds = np.argmax(predictions, axis=2)`\n",
    "\n",
    "\n",
    "### When I print `eval_pred.label_ids` - I get below a 3D-Tensor\n",
    "\n",
    "```\n",
    "[[-100    3 -100 ... -100 -100 -100]\n",
    " [-100    0 -100 ... -100 -100 -100]\n",
    " [-100    0    0 ... -100 -100 -100]\n",
    "  ...\n",
    " [-100    0    0 ... -100 -100 -100]\n",
    " [-100    0    0 ... -100 -100 -100]\n",
    " [-100    0    0 ... -100 -100 -100]]\n",
    "\n",
    "```\n",
    "\n",
    "In above each list corresponding to a single example in our validation or test sets. \n",
    "\n",
    "------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "id": "ajZ6-Nel3pUm"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def generate_list_for_compute_metrics(predictions, label_ids):\n",
    "    \"\"\"\n",
    "    Function to generate prediction and true labels lists for computing metrics.\n",
    "\n",
    "    Parameters:\n",
    "    predictions (np.ndarray): A 2D numpy array containing the predicted label IDs for each token in each example.\n",
    "    label_ids (np.ndarray): A 2D numpy array containing the true label IDs for each token in each example.\n",
    "\n",
    "    Returns:\n",
    "    preds_labels_list (list): A list of lists, where each sublist contains the predicted labels for each token in an example.\n",
    "    true_labels_list (list): A list of lists, where each sublist contains the true labels for each token in an example.\n",
    "    \"\"\"\n",
    "    # Get the predicted labels by taking the argmax over the second dimension of the predictions array\n",
    "    preds = np.argmax(predictions, axis=2)\n",
    "    batch_size, seq_len = preds.shape\n",
    "    preds_labels_list, true_labels_list = [], []\n",
    "\n",
    "    # Iterate over each example in the batch\n",
    "    for batch_idx in range(batch_size):\n",
    "        example_labels, example_preds = [], []\n",
    "        # Iterate over each token in the example\n",
    "        for seq_idx in range(seq_len):\n",
    "            # Ignore tokens with label ID = -100 (these are special tokens or subwords that we masked during training)\n",
    "            if label_ids[batch_idx, seq_idx] != -100:\n",
    "                # Append the predicted and true labels for the token to the lists for the current example\n",
    "                example_preds.append(index2tag[preds[batch_idx][seq_idx]])\n",
    "                example_labels.append(index2tag[label_ids[batch_idx][seq_idx]])\n",
    "        # Append the lists for the current example to the main lists\n",
    "        preds_labels_list.append(example_preds)\n",
    "        true_labels_list.append(example_labels)\n",
    "\n",
    "    # Return the lists of predicted and true labels\n",
    "    return preds_labels_list, true_labels_list\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "id": "Om3qCKQu3pUn"
   },
   "outputs": [],
   "source": [
    "\n",
    "num_epochs = 3\n",
    "batch_size = 24\n",
    "logging_steps = len(panx_de_encoded[\"train\"]) // batch_size\n",
    "model_name = f\"{xlmr_model_name}-finetuned-panx-de\"\n",
    "\n",
    "\n",
    "training_args = TrainingArguments(\n",
    "    output_dir=model_name, log_level=\"error\", num_train_epochs=num_epochs, \n",
    "    per_device_train_batch_size=batch_size, \n",
    "    per_device_eval_batch_size=batch_size, evaluation_strategy=\"epoch\", \n",
    "    save_steps=1e6, weight_decay=0.01, disable_tqdm=False, \n",
    "    logging_steps=logging_steps, push_to_hub=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "id": "4IB2rl9c3pUn"
   },
   "outputs": [],
   "source": [
    "\n",
    "# from huggingface_hub import notebook_login\n",
    "\n",
    "# notebook_login()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "id": "whwZeE0O3pUn"
   },
   "outputs": [],
   "source": [
    "from seqeval.metrics import f1_score\n",
    "from transformers import EvalPrediction\n",
    "\n",
    "''' compute_metrics (Callable[[EvalPrediction], Dict], optional) – The function that will be used to compute metrics at evaluation. Must take a EvalPrediction and return a dictionary string to metric values. \n",
    "https://huggingface.co/transformers/v4.2.2/main_classes/trainer.html#id1\n",
    "'''\n",
    "def compute_metrics(eval_pred: EvalPrediction):\n",
    "    \"\"\"\n",
    "    Computes the F1 score metric for the model evaluation.\n",
    "\n",
    "    Args:\n",
    "        eval_pred (EvalPrediction): Object containing the model predictions and labels.\n",
    "\n",
    "    Returns:\n",
    "        dict: Dictionary containing the computed F1 score.\n",
    "\n",
    "    \"\"\"\n",
    "    # print('eval_pred ', eval_pred)\n",
    "    print('eval_pred.predictions ', eval_pred.predictions)\n",
    "    # Output =>  <transformers.trainer_utils.EvalPrediction object\n",
    "    y_pred, y_true = generate_list_for_compute_metrics(eval_pred.predictions, \n",
    "                                       eval_pred.label_ids)\n",
    "    return {\"f1\": f1_score(y_true, y_pred)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "id": "X6qAxpma3pUo"
   },
   "outputs": [],
   "source": [
    "from transformers import DataCollatorForTokenClassification\n",
    "\n",
    "data_collator = DataCollatorForTokenClassification(xlmr_tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "id": "r04p4n4H3pUo"
   },
   "outputs": [],
   "source": [
    "def model_init():\n",
    "    return (XLMRobertaForTokenClassification\n",
    "            .from_pretrained(xlmr_model_name, config=xlmr_config)\n",
    "            .to(device))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "id": "R43thSZC3pUo"
   },
   "outputs": [],
   "source": [
    "#hide\n",
    "# %env TOKENIZERS_PARALLELISM=false"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-Tuning XLM-RoBERTa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "id": "RQGB1SCH3pUo"
   },
   "outputs": [],
   "source": [
    "\n",
    "from transformers import Trainer\n",
    "\n",
    "trainer = Trainer(model_init=model_init, args=training_args, \n",
    "                  data_collator=data_collator, compute_metrics=compute_metrics,\n",
    "                  train_dataset=panx_de_encoded[\"train\"],\n",
    "                  eval_dataset=panx_de_encoded[\"validation\"], \n",
    "                  tokenizer=xlmr_tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "Oq0jozQC3pUp",
    "outputId": "f4ba7daf-393e-4758-f008-f6e66dbbab39"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:310: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  FutureWarning,\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='1575' max='1575' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [1575/1575 08:47, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.255300</td>\n",
       "      <td>0.157545</td>\n",
       "      <td>0.827877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.128400</td>\n",
       "      <td>0.138613</td>\n",
       "      <td>0.846321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.081300</td>\n",
       "      <td>0.136510</td>\n",
       "      <td>0.864874</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 5.5185753e-01 -2.7944833e-01 -7.5236917e-01 ...  3.5712823e-01\n",
      "   -6.6847444e-01  2.7783272e-01]\n",
      "  [-6.0417420e-01  1.7045190e+00 -3.0190976e+00 ... -8.1113666e-01\n",
      "    1.5366187e+00 -3.7904561e+00]\n",
      "  [-5.0854206e-01 -1.9779130e+00  9.6103489e-02 ...  4.6591806e+00\n",
      "   -1.7985896e+00 -6.7350179e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.6376225e-01 -1.5263814e-01 -4.9990118e-01 ...  8.7990403e-02\n",
      "   -6.6905731e-01  3.3763283e-01]\n",
      "  [ 8.2456388e+00 -1.9777597e+00 -2.1865675e+00 ... -3.8966548e-01\n",
      "   -1.6170585e+00 -1.6310189e+00]\n",
      "  [ 8.1376953e+00 -2.0120018e+00 -1.8712316e+00 ... -2.0535491e-01\n",
      "   -1.6027997e+00 -1.3040750e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 2.1459823e+00 -4.7410148e-01 -1.4798362e+00 ...  1.6768871e-02\n",
      "   -4.4227758e-01 -2.7634958e-02]\n",
      "  [ 8.1064272e+00 -2.1034284e+00 -1.8258690e+00 ... -4.9337369e-01\n",
      "   -1.6840947e+00 -1.3385955e+00]\n",
      "  [ 8.2820921e+00 -2.4913888e+00 -1.6583630e+00 ... -2.0041099e-01\n",
      "   -2.0127506e+00 -1.1330633e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 3.2323623e+00 -9.3965220e-01 -1.6234053e+00 ...  4.3480162e-02\n",
      "   -3.3202305e-01  6.3045509e-02]\n",
      "  [ 8.1087055e+00 -2.3764012e+00 -1.8841689e+00 ... -1.0439133e-01\n",
      "   -1.7752892e+00 -1.3277261e+00]\n",
      "  [ 8.2374811e+00 -2.4733608e+00 -1.8725168e+00 ... -5.8374520e-02\n",
      "   -1.9236329e+00 -1.1627105e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 2.4286549e+00 -4.5357552e-01 -1.6082292e+00 ...  9.2238200e-01\n",
      "   -7.4052751e-01 -3.5676041e-01]\n",
      "  [ 8.0177279e+00 -2.2603626e+00 -1.8107536e+00 ... -2.2976352e-03\n",
      "   -1.8939985e+00 -1.4457127e+00]\n",
      "  [ 7.1001148e+00 -2.0324936e+00 -2.0306261e+00 ...  2.8527951e-01\n",
      "   -1.5444032e+00 -1.8935751e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 8.9968073e-01 -2.1168074e-01 -8.7151623e-01 ... -7.1001701e-02\n",
      "   -4.9541888e-01  2.6229987e-01]\n",
      "  [ 8.0508490e+00 -2.2282817e+00 -1.8727332e+00 ... -6.4819962e-01\n",
      "   -1.8835450e+00 -1.2872565e+00]\n",
      "  [ 8.0300722e+00 -2.0890326e+00 -1.8690028e+00 ... -8.1332827e-01\n",
      "   -1.8503753e+00 -1.4855906e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n",
      "eval_pred.predictions  [[[ 4.8686886e-01 -2.8893262e-01 -6.9235235e-01 ...  3.4208387e-01\n",
      "   -6.9075578e-01  2.6379955e-01]\n",
      "  [-1.3569305e+00  1.9009511e+00 -2.5981908e+00 ... -1.2650105e+00\n",
      "    2.0003865e+00 -3.4807880e+00]\n",
      "  [-1.8406988e+00 -2.2236323e+00  1.2167749e+00 ...  4.7730665e+00\n",
      "   -1.7364494e+00 -3.4666207e-02]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 8.8021517e-01 -1.8434227e-01 -6.2004304e-01 ...  1.3712670e-01\n",
      "   -8.3892453e-01  4.8777699e-01]\n",
      "  [ 8.5467300e+00 -2.0044749e+00 -2.3894937e+00 ... -1.0395962e+00\n",
      "   -1.4939209e+00 -1.5914491e+00]\n",
      "  [ 8.4867687e+00 -2.1121356e+00 -2.0668008e+00 ... -9.3017077e-01\n",
      "   -1.6000111e+00 -1.1217020e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.8084009e+00 -5.0744158e-01 -2.0759974e+00 ... -6.7363942e-01\n",
      "   -4.9682114e-01 -6.5478939e-01]\n",
      "  [ 8.5046167e+00 -2.0271738e+00 -1.9099694e+00 ... -1.2670538e+00\n",
      "   -1.6317055e+00 -1.3193947e+00]\n",
      "  [ 8.8503418e+00 -2.4506328e+00 -1.8808270e+00 ... -9.9626660e-01\n",
      "   -2.0708957e+00 -1.0812110e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 4.4861546e+00 -7.4159312e-01 -2.1670775e+00 ... -8.8218868e-01\n",
      "   -5.2893090e-01 -7.3173279e-01]\n",
      "  [ 8.7941113e+00 -2.3307762e+00 -2.2064826e+00 ... -9.7156787e-01\n",
      "   -1.7915744e+00 -1.3626432e+00]\n",
      "  [ 8.8430395e+00 -2.3396518e+00 -2.2427485e+00 ... -9.5813769e-01\n",
      "   -1.9694481e+00 -1.2386004e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 2.5713673e+00 -2.5329813e-01 -1.4463087e+00 ... -2.3766296e-01\n",
      "   -7.5478029e-01 -4.1908514e-01]\n",
      "  [ 8.5398245e+00 -1.9623322e+00 -1.9964058e+00 ... -8.8193679e-01\n",
      "   -1.7762251e+00 -1.6864003e+00]\n",
      "  [ 7.1626549e+00 -1.7526901e+00 -1.9145701e+00 ... -1.3317175e-01\n",
      "   -1.4103826e+00 -2.1190956e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 9.2466116e-01 -1.7971665e-01 -9.1676641e-01 ... -1.6550805e-01\n",
      "   -4.9890763e-01  1.7296284e-01]\n",
      "  [ 8.2154541e+00 -2.0737975e+00 -1.8645430e+00 ... -1.2538604e+00\n",
      "   -1.8862159e+00 -1.3854374e+00]\n",
      "  [ 8.2854748e+00 -2.1641505e+00 -1.4808004e+00 ... -1.2259088e+00\n",
      "   -2.2024603e+00 -1.2073592e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n",
      "eval_pred.predictions  [[[ 4.2822057e-01 -2.4668892e-01 -6.6056097e-01 ...  4.8391807e-01\n",
      "   -7.7924144e-01  1.9661833e-01]\n",
      "  [-1.6177338e+00  2.1463876e+00 -3.0167634e+00 ... -1.2209541e+00\n",
      "    1.4632130e+00 -3.7305620e+00]\n",
      "  [-2.0813432e+00 -1.7880467e+00  8.9826584e-01 ...  4.5865231e+00\n",
      "   -1.6231425e+00 -6.8946230e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.3092749e-01  1.9917661e-01 -1.7368760e-02 ... -4.2736679e-02\n",
      "   -9.4946951e-01  2.7366027e-01]\n",
      "  [ 9.0695515e+00 -2.1953573e+00 -2.2917879e+00 ... -1.1032797e+00\n",
      "   -1.5802649e+00 -1.6272942e+00]\n",
      "  [ 9.0205145e+00 -2.2079537e+00 -2.0486577e+00 ... -1.0813054e+00\n",
      "   -1.6243299e+00 -1.3591262e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.2048941e+00 -5.2996403e-01 -1.8565143e+00 ... -9.5477027e-01\n",
      "    3.8208190e-01 -1.4136621e+00]\n",
      "  [ 8.6292334e+00 -2.0559981e+00 -1.9298774e+00 ... -1.3581136e+00\n",
      "   -1.4508513e+00 -1.5307190e+00]\n",
      "  [ 9.1498423e+00 -2.4832242e+00 -1.8504959e+00 ... -1.1084143e+00\n",
      "   -1.9210169e+00 -1.3353750e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 6.0000014e+00 -9.5296836e-01 -2.4098272e+00 ... -1.4608077e+00\n",
      "   -8.1508107e-02 -1.3898863e+00]\n",
      "  [ 9.0262184e+00 -2.3442941e+00 -2.0762224e+00 ... -1.0964475e+00\n",
      "   -1.6675217e+00 -1.5385301e+00]\n",
      "  [ 9.2396870e+00 -2.3909128e+00 -2.1913469e+00 ... -1.1529007e+00\n",
      "   -1.7923570e+00 -1.5112613e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.9747972e+00 -4.4678491e-01 -2.1053622e+00 ... -5.0252223e-01\n",
      "   -6.4804488e-01 -1.4826046e+00]\n",
      "  [ 8.4725256e+00 -2.0049002e+00 -1.8329651e+00 ... -8.7825662e-01\n",
      "   -1.7133546e+00 -1.8380109e+00]\n",
      "  [ 7.6252661e+00 -1.6712410e+00 -2.3585730e+00 ... -4.9015585e-01\n",
      "   -1.2003969e+00 -2.6005507e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 8.7560129e-01 -1.5182243e-01 -8.7084842e-01 ... -2.0111658e-01\n",
      "   -4.9599919e-01  1.7103206e-01]\n",
      "  [ 8.9191275e+00 -2.0267656e+00 -1.6877822e+00 ... -1.6239527e+00\n",
      "   -1.8704675e+00 -1.6193720e+00]\n",
      "  [ 8.8970709e+00 -1.9686152e+00 -1.5114130e+00 ... -1.6269511e+00\n",
      "   -1.9776002e+00 -1.6775986e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=1575, training_loss=0.15484960851215182, metrics={'train_runtime': 528.0181, 'train_samples_per_second': 71.475, 'train_steps_per_second': 2.983, 'total_flos': 863012377186080.0, 'train_loss': 0.15484960851215182, 'epoch': 3.0})"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#hide_input\n",
    "# THIS WILL NOT RUN IN LOCAL MACHINE WITHOUT GPU\n",
    "trainer.train()\n",
    "# trainer.push_to_hub(commit_message=\"Training completed!\") # PAUL - COMMENTING OUT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 143
    },
    "id": "TFeiwO_v3pUp",
    "outputId": "1a311cfa-17c9-45f5-dfbd-cf9721ee26dc"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-89dda73d-3f0b-492d-8d2a-896c265400c0\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.2553</td>\n",
       "      <td>0.157545</td>\n",
       "      <td>0.827877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0.1284</td>\n",
       "      <td>0.138613</td>\n",
       "      <td>0.846321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>0.0813</td>\n",
       "      <td>0.136510</td>\n",
       "      <td>0.864874</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-89dda73d-3f0b-492d-8d2a-896c265400c0')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-89dda73d-3f0b-492d-8d2a-896c265400c0 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-89dda73d-3f0b-492d-8d2a-896c265400c0');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "   Epoch  Training Loss  Validation Loss        F1\n",
       "0      1         0.2553         0.157545  0.827877\n",
       "2      2         0.1284         0.138613  0.846321\n",
       "4      3         0.0813         0.136510  0.864874"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# hide_input\n",
    "df = pd.DataFrame(trainer.state.log_history)[['epoch','loss' ,'eval_loss', 'eval_f1']]\n",
    "df = df.rename(columns={\"epoch\":\"Epoch\",\"loss\": \"Training Loss\", \"eval_loss\": \"Validation Loss\", \"eval_f1\":\"F1\"})\n",
    "df['Epoch'] = df[\"Epoch\"].apply(lambda x: round(x))\n",
    "df['Training Loss'] = df[\"Training Loss\"].ffill()\n",
    "df[['Validation Loss', 'F1']] = df[['Validation Loss', 'F1']].bfill().ffill()\n",
    "df.drop_duplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 181
    },
    "id": "eFYW9I_v3pUp",
    "outputId": "4b222661-20f0-42b5-ce17-707302053812"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_ids  tensor([[     0,    622,  35473,   4932,   1256,  22667,  47582, 105173,     23,\n",
      "          37061,      2]], device='cuda:0')\n",
      "Shape of outputs: torch.Size([1, 11, 7])\n",
      "predictions  tensor([[0, 0, 3, 4, 0, 0, 0, 0, 0, 5, 0]], device='cuda:0')\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-8103c6e3-27e5-4258-83cf-3e1fb0474433\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Tokens</th>\n",
       "      <td>&lt;s&gt;</td>\n",
       "      <td>▁Die</td>\n",
       "      <td>▁Deutsche</td>\n",
       "      <td>▁Bank</td>\n",
       "      <td>▁hat</td>\n",
       "      <td>▁ihren</td>\n",
       "      <td>▁Haupt</td>\n",
       "      <td>sitz</td>\n",
       "      <td>▁in</td>\n",
       "      <td>▁Frankfurt</td>\n",
       "      <td>&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tags</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8103c6e3-27e5-4258-83cf-3e1fb0474433')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-8103c6e3-27e5-4258-83cf-3e1fb0474433 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-8103c6e3-27e5-4258-83cf-3e1fb0474433');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "         0     1          2      3     4       5       6     7    8   \\\n",
       "Tokens  <s>  ▁Die  ▁Deutsche  ▁Bank  ▁hat  ▁ihren  ▁Haupt  sitz  ▁in   \n",
       "Tags      O     O      B-ORG  I-ORG     O       O       O     O    O   \n",
       "\n",
       "                9     10  \n",
       "Tokens  ▁Frankfurt  </s>  \n",
       "Tags         B-LOC     O  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "# text_de = \"Jeff Dean ist ein Informatiker bei Google in Kalifornien\"\n",
    "\n",
    "# The english of the below random German sentence is \n",
    "# Deutsche Bank is headquartered in Frankfurt\n",
    "\n",
    "text_de = \"Die Deutsche Bank hat ihren Hauptsitz in Frankfurt\"\n",
    "tag_text(text_de, tags, trainer.model, xlmr_tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "g_kCCMpD3pUq",
    "outputId": "e3192367-c428-4a14-9103-3de058dd44d8"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "\"\\nDataset({\\n    features: ['input_ids', 'attention_mask', 'labels'],\\n    num_rows: 6290\\n})\\n\""
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_set_batch = panx_de_encoded[\"validation\"]\n",
    "valid_set_batch\n",
    "\n",
    "'''\n",
    "Dataset({\n",
    "    features: ['input_ids', 'attention_mask', 'labels'],\n",
    "    num_rows: 6290\n",
    "})\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SPsE3oxt3pUq",
    "outputId": "7183311a-70b9-4bb2-e29d-e51983a4e5cc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),\n",
       " 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None),\n",
       " 'labels': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)}"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_set_batch.features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "HCtzN0ym3pUq",
    "outputId": "13e2d9ea-153b-4ab2-ffa0-a1c6d5440da0"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "\" \\n{'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),\\n 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None),\\n 'labels': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)}\\n\""
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_set_batch.features[\"input_ids\"]\n",
    "\n",
    "''' \n",
    "{'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),\n",
    " 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None),\n",
    " 'labels': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)}\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "RzlJCmFd3pUq",
    "outputId": "8f88fb68-cf06-429b-9650-6c365c57f5c4"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "\" Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None) \""
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_set_batch.features[\"attention_mask\"]\n",
    "''' Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None) '''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "id": "EgyhFca53pUq"
   },
   "outputs": [],
   "source": [
    "from torch.nn.functional import cross_entropy\n",
    "\n",
    "def get_forward_loss_and_label(batch):\n",
    "    \"\"\"\n",
    "    Function to compute the loss and the predicted label for a given batch.\n",
    "\n",
    "    Parameters:\n",
    "    batch (dict): A batch of input data, containing the input_ids, attention_mask, and labels.\n",
    "\n",
    "    Returns:\n",
    "    results (dict): A dictionary containing the loss and predicted labels.\n",
    "    \"\"\"\n",
    "    # Convert dict of lists to list of dicts suitable for data collator\n",
    "    features = [dict(zip(batch, t)) for t in zip(*batch.values())]\n",
    "    # Pad inputs and labels and put all tensors on device\n",
    "    batch = data_collator(features)\n",
    "    input_ids = batch[\"input_ids\"].to(device)\n",
    "    attention_mask = batch[\"attention_mask\"].to(device)\n",
    "    labels = batch[\"labels\"].to(device)\n",
    "    with torch.no_grad():\n",
    "        # Pass data through model  \n",
    "        # trainer.model(PreTrainedModel or torch.nn.Module, optional)\n",
    "        # This is the model to train, evaluate or use for predictions.\n",
    "        output = trainer.model(input_ids, attention_mask)\n",
    "        # Logit.size: [batch_size, sequence_length, classes]\n",
    "        # Predict class with largest logit value on classes axis\n",
    "        predicted_label = torch.argmax(output.logits, axis=-1).cpu().numpy()\n",
    "    # Calculate loss per token after flattening batch dimension with view\n",
    "    loss = cross_entropy(output.logits.view(-1, 7), \n",
    "                         labels.view(-1), reduction=\"none\")\n",
    "    # Unflatten batch dimension and convert to numpy array\n",
    "    loss = loss.view(len(input_ids), -1).cpu().numpy()\n",
    "\n",
    "    return {\"loss\":loss, \"predicted_label\": predicted_label}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 86,
     "referenced_widgets": [
      "fdb2de29b6dd4e6fbd4d790b97c8fd9c",
      "060a8a505aab4fe7885611e30b6bbe37",
      "889691d5397a4e07acbbc3c3b6db7561",
      "77512d0ba25844488753e322cc121907",
      "e419c224a82b4ab4a09271dee9c22df2",
      "83ac07d90ca94345aa20146b0a9c3edf",
      "f1163b0f9eea40f8b086b18eda7b1e87",
      "55022f8c3a7649ceb6d335a7d830bd28",
      "1565cda3d181427f8f2d38d6316a0e34",
      "423d5c3212154705a1fd85851515c9f0",
      "9129785f192e46179d7a8faaa30b7f0a"
     ]
    },
    "id": "dUSgxbSq3pUr",
    "outputId": "00e12cf6-0525-4c83-ee74-f118a60364e4"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parameter 'function'=<function get_forward_loss_and_label at 0x7f432c13e7a0> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fdb2de29b6dd4e6fbd4d790b97c8fd9c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/197 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "valid_set_with_loss = valid_set_batch.map(get_forward_loss_and_label, batched=True, batch_size=32)\n",
    "\n",
    "df = valid_set_with_loss.to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rYxudVQ13pUr",
    "outputId": "eb4c4182-86fd-4368-bd54-9d52e182a67d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6290, 5)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 337
    },
    "id": "dhqc--mD3pUr",
    "outputId": "78e36631-2fa6-4529-bf80-ba8f07987edc"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-c609507e-a8a5-43db-a019-aa628fd5de08\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input_ids</th>\n",
       "      <th>attention_mask</th>\n",
       "      <th>labels</th>\n",
       "      <th>loss</th>\n",
       "      <th>predicted_label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[0, 10699, 11, 15, 16104, 1388, 2]</td>\n",
       "      <td>[1, 1, 1, 1, 1, 1, 1]</td>\n",
       "      <td>[-100, 3, -100, 4, 4, 4, -100]</td>\n",
       "      <td>[0.0, 0.028868396, 0.0, 0.06327692, 0.03726901...</td>\n",
       "      <td>[4, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[0, 56530, 25216, 30121, 152385, 19229, 83982,...</td>\n",
       "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "      <td>[-100, 0, -100, -100, -100, -100, 3, -100, -10...</td>\n",
       "      <td>[0.0, 0.00014768941, 0.0, 0.0, 0.0, 0.0, 2.922...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 1, 2, 2, 2, 2, 2, 2, 2, 2, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[0, 159093, 165, 38506, 122, 153080, 29088, 57...</td>\n",
       "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]</td>\n",
       "      <td>[-100, 0, 0, 0, 0, 3, -100, -100, 0, -100, 0, ...</td>\n",
       "      <td>[0.0, 0.0002630603, 0.00012516193, 0.000276884...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[0, 16459, 242, 5106, 6, 198715, 5106, 242, 2]</td>\n",
       "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1]</td>\n",
       "      <td>[-100, 0, 0, 0, 5, -100, 0, 0, -100]</td>\n",
       "      <td>[0.0, 0.00018630676, 0.00014995404, 0.00019464...</td>\n",
       "      <td>[0, 0, 0, 0, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[0, 11022, 2315, 7418, 1079, 8186, 57242, 97, ...</td>\n",
       "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "      <td>[-100, 0, 0, 0, 0, 0, 0, 0, -100, 0, 0, 0, 3, ...</td>\n",
       "      <td>[0.0, 0.0001859492, 0.00014983486, 0.000167951...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 4, 4, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c609507e-a8a5-43db-a019-aa628fd5de08')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-c609507e-a8a5-43db-a019-aa628fd5de08 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-c609507e-a8a5-43db-a019-aa628fd5de08');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                                           input_ids  \\\n",
       "0                 [0, 10699, 11, 15, 16104, 1388, 2]   \n",
       "1  [0, 56530, 25216, 30121, 152385, 19229, 83982,...   \n",
       "2  [0, 159093, 165, 38506, 122, 153080, 29088, 57...   \n",
       "3     [0, 16459, 242, 5106, 6, 198715, 5106, 242, 2]   \n",
       "4  [0, 11022, 2315, 7418, 1079, 8186, 57242, 97, ...   \n",
       "\n",
       "                                      attention_mask  \\\n",
       "0                              [1, 1, 1, 1, 1, 1, 1]   \n",
       "1  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...   \n",
       "2         [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]   \n",
       "3                        [1, 1, 1, 1, 1, 1, 1, 1, 1]   \n",
       "4  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...   \n",
       "\n",
       "                                              labels  \\\n",
       "0                     [-100, 3, -100, 4, 4, 4, -100]   \n",
       "1  [-100, 0, -100, -100, -100, -100, 3, -100, -10...   \n",
       "2  [-100, 0, 0, 0, 0, 3, -100, -100, 0, -100, 0, ...   \n",
       "3               [-100, 0, 0, 0, 5, -100, 0, 0, -100]   \n",
       "4  [-100, 0, 0, 0, 0, 0, 0, 0, -100, 0, 0, 0, 3, ...   \n",
       "\n",
       "                                                loss  \\\n",
       "0  [0.0, 0.028868396, 0.0, 0.06327692, 0.03726901...   \n",
       "1  [0.0, 0.00014768941, 0.0, 0.0, 0.0, 0.0, 2.922...   \n",
       "2  [0.0, 0.0002630603, 0.00012516193, 0.000276884...   \n",
       "3  [0.0, 0.00018630676, 0.00014995404, 0.00019464...   \n",
       "4  [0.0, 0.0001859492, 0.00014983486, 0.000167951...   \n",
       "\n",
       "                                     predicted_label  \n",
       "0  [4, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, ...  \n",
       "1  [0, 0, 0, 0, 0, 0, 1, 2, 2, 2, 2, 2, 2, 2, 2, ...  \n",
       "2  [0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  \n",
       "3  [0, 0, 0, 0, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  \n",
       "4  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 4, 4, ...  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross-Lingual Transfer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "id": "pmE3g3mT3pUs"
   },
   "outputs": [],
   "source": [
    "def get_f1_score(trainer, dataset):\n",
    "    return trainer.predict(dataset).metrics[\"test_f1\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "ZgLZ6is-3pUs",
    "outputId": "e5af6fa0-0a4a-4004-f965-9152970648a0"
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 1.7765553e+00 -9.3398362e-01 -1.1353534e+00 ... -2.3706818e-01\n",
      "   -2.9669267e-01  1.6833218e+00]\n",
      "  [ 8.9977589e+00 -2.1211178e+00 -2.3355274e+00 ... -1.2288058e+00\n",
      "   -1.4406507e+00 -1.7622006e+00]\n",
      "  [ 8.9387512e+00 -2.1752007e+00 -2.1381011e+00 ... -1.1254311e+00\n",
      "   -1.5119449e+00 -1.5425693e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.5588489e+00  5.1431793e-01 -1.1737152e+00 ... -2.4808837e-01\n",
      "   -1.4566694e+00 -1.6115543e+00]\n",
      "  [ 8.8124352e+00 -1.9399652e+00 -1.7310735e+00 ... -1.0168083e+00\n",
      "   -1.9551457e+00 -1.7452103e+00]\n",
      "  [-2.3937516e+00  6.3138485e+00 -1.7222172e+00 ... -1.1291534e+00\n",
      "   -8.8868713e-01 -3.5382648e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.9817972e+00 -2.0532893e-02 -1.1766983e+00 ... -2.9811943e-01\n",
      "   -8.1727958e-01 -3.9496076e-01]\n",
      "  [ 8.9302244e+00 -2.2610412e+00 -1.8214946e+00 ... -1.3196059e+00\n",
      "   -1.8493071e+00 -1.5847571e+00]\n",
      "  [ 8.3512716e+00 -1.6467820e+00 -1.6653360e+00 ... -1.6216062e+00\n",
      "   -1.5056630e+00 -1.8035702e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 4.1670470e+00 -6.4084685e-01 -2.0444140e+00 ... -1.1725737e-01\n",
      "   -7.4682850e-01 -9.3917942e-01]\n",
      "  [ 8.6661863e+00 -1.9436055e+00 -1.8757805e+00 ... -1.1640029e+00\n",
      "   -1.4608557e+00 -1.7642066e+00]\n",
      "  [ 8.9185133e+00 -2.1223247e+00 -1.9691528e+00 ... -1.4010717e+00\n",
      "   -1.3870869e+00 -1.6586138e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 5.3744092e+00 -1.2626587e+00 -2.2955709e+00 ... -8.4587687e-01\n",
      "   -2.8115982e-01 -9.2782778e-01]\n",
      "  [ 9.0833416e+00 -2.3162589e+00 -2.1190715e+00 ... -1.1551871e+00\n",
      "   -1.7744144e+00 -1.5317336e+00]\n",
      "  [ 6.5021224e+00 -2.0300446e+00 -3.3669038e+00 ... -1.7886679e+00\n",
      "    1.1301590e+00 -1.0347224e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.1733800e+00 -5.0175184e-01 -1.0581484e+00 ... -3.1613961e-01\n",
      "   -2.3688151e-01  5.8968866e-01]\n",
      "  [-1.7105892e+00 -9.6579558e-01 -2.8940816e+00 ... -1.0312631e+00\n",
      "    6.9309163e+00 -7.0580447e-01]\n",
      "  [-2.3039618e+00 -3.1188338e+00  6.3475229e-02 ...  1.4907547e+00\n",
      "    4.9641529e-01  6.5458922e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n",
      "F1-score of [de] model on [de] dataset: 0.869\n"
     ]
    }
   ],
   "source": [
    "f1_scores = defaultdict(dict)\n",
    "f1_scores[\"de\"][\"de\"] = get_f1_score(trainer, panx_de_encoded[\"test\"])\n",
    "print(f\"F1-score of [de] model on [de] dataset: {f1_scores['de']['de']:.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 181
    },
    "id": "U98_4ncl3pUs",
    "outputId": "962d0822-c97d-4143-b958-4692e822d55c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_ids  tensor([[     0, 101681,    262,     66,    437, 124588,   6594,  19998,   1815,\n",
      "             22,  25971,   2472,    701,      2]], device='cuda:0')\n",
      "Shape of outputs: torch.Size([1, 14, 7])\n",
      "predictions  tensor([[0, 1, 2, 2, 0, 0, 0, 0, 3, 0, 5, 6, 6, 0]], device='cuda:0')\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-ca4bd82a-7699-44d7-944d-0339411db64a\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Tokens</th>\n",
       "      <td>&lt;s&gt;</td>\n",
       "      <td>▁Jeff</td>\n",
       "      <td>▁De</td>\n",
       "      <td>an</td>\n",
       "      <td>▁est</td>\n",
       "      <td>▁informatic</td>\n",
       "      <td>ien</td>\n",
       "      <td>▁chez</td>\n",
       "      <td>▁Google</td>\n",
       "      <td>▁en</td>\n",
       "      <td>▁Cali</td>\n",
       "      <td>for</td>\n",
       "      <td>nie</td>\n",
       "      <td>&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tags</th>\n",
       "      <td>O</td>\n",
       "      <td>B-PER</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ca4bd82a-7699-44d7-944d-0339411db64a')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-ca4bd82a-7699-44d7-944d-0339411db64a button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-ca4bd82a-7699-44d7-944d-0339411db64a');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "         0      1      2      3     4            5    6      7        8    9   \\\n",
       "Tokens  <s>  ▁Jeff    ▁De     an  ▁est  ▁informatic  ien  ▁chez  ▁Google  ▁en   \n",
       "Tags      O  B-PER  I-PER  I-PER     O            O    O      O    B-ORG    O   \n",
       "\n",
       "           10     11     12    13  \n",
       "Tokens  ▁Cali    for    nie  </s>  \n",
       "Tags    B-LOC  I-LOC  I-LOC     O  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_fr = \"Jeff Dean est informaticien chez Google en Californie\"\n",
    "tag_text(text_fr, tags, trainer.model, xlmr_tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "id": "IeGiK_K73pUs"
   },
   "outputs": [],
   "source": [
    "def evaluate_lang_performance(lang, trainer):\n",
    "    panx_ds = encode_panx_dataset(panx_ds_combined[lang])\n",
    "    return get_f1_score(trainer, panx_ds[\"test\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "referenced_widgets": [
      "8e1b38886d4843988814fc8f66953808",
      "b2202c1db1d94ae68de1e7dd25b652b8",
      "f719f7fdcc894ceea99ec78d7b84fbb7",
      "e56882ac1c034cb49be257285fc1bf9d",
      "c306ab035b4e44a8bc1f1c813b224f74",
      "12b5cb91b9a645e588235d3573cf7b31",
      "01740edabff446a0935dd097dbf0f295",
      "5e7fa665ba56418ca351e9d049901aa4",
      "f636b121dd694106be6b7938f3bd677c",
      "ae5a2a816a654ff79f40ed0c21dd86ba",
      "4afd7d35ce6e421dbc4fe67a2e59b061",
      "448a446fa3c34b73b9335c9e831a8e5e",
      "dfe170856aad4b0cb8ab484c1bb9044d",
      "5e2aa9f014ac46e8993fa740ddeecdc5",
      "87a442a80a18410aa1a8bd8b9a4fd763",
      "80dfcdb2f016497f83055781655c3a4e",
      "47864d588d8e40e6a5127b4c1ca8e9da",
      "c78295b5fa504243b898cea0f510c3ed",
      "5659cde4a272492fb14dda55400d6ef5",
      "3157930ac9eb4b35bb2e72e54bd4410a",
      "ab4d3e6b79234a6a904e78425868c199",
      "c7671fe58c79492dae2ec41dca7c6d9b",
      "176379cc286e4caabcdee0ae540b9f34",
      "84f6e54498d545bb9d41df8f75aaf8c2",
      "cfa6b5b58c8949e2b58a749269499fb7",
      "e3096f901a644a439634bf609faded4f",
      "f759ed05d93942899a1852af8c308824",
      "45f2ccb89613456a87b4f0b17e444f49",
      "b62fb85bcd9846f391016e02911597d8",
      "430d63c852b941ba9a9174744d865d8a",
      "f55e19b106814d34ab74ba2e908aff2c",
      "fc29df0299814ab489deeb0e1fd9f89f",
      "273adb7bced045bd8321846e07746ab0"
     ]
    },
    "id": "Jl8JU9xb3pUt",
    "outputId": "c2b3b42e-8a78-4eda-df54-8abaf2b00928"
   },
   "outputs": [],
   "source": [
    "\n",
    "f1_scores[\"de\"][\"fr\"] = evaluate_lang_performance(\"fr\", trainer)\n",
    "print(f\"F1-score of [de] model on [fr] dataset: {f1_scores['de']['fr']:.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BeUqmjvL3pUt",
    "outputId": "680d8787-2bd5-4cd7-eefb-1f4f819f605d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1-score of [de] model on [fr] dataset: 0.702\n"
     ]
    }
   ],
   "source": [
    "# hide_input\n",
    "print(f\"F1-score of [de] model on [fr] dataset: {f1_scores['de']['fr']:.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "referenced_widgets": [
      "3a893b9431744117ac9b7e50c2a9f013",
      "622cab19793f4f92910f10b755d30e52",
      "90f1a2b774fe4c8d8c795b1b76f5fce2",
      "ed920a6c4e8c4147a6f2ed98e08d64bb",
      "299b8e07b33741e79ca4ec8d4e3b5f9f",
      "03a260f79a0e4f32b46698d959f22078",
      "d0e33ec011824dd6a25f49efa7a22fe6",
      "6ded3cca879246e7a286ab3e47ddc518",
      "5302b322abee472c8585944114d445ad",
      "675d8eebafc84f2093659df4a93f0180",
      "f71aa4ec35b346a684ff9f90bb7ea04d",
      "df272f940b3341bdb5eb15b8084b11af",
      "289a2a99024741808313f44980ee2efc",
      "cb7e6845f9024b91aa4fed9f463db6d2",
      "648a592e0eeb4ee4b57365b5dcf8c703",
      "f26e946754a14424a5c686c3177e9ed8",
      "25d86f7f215744d09a83e3986fcf3c20",
      "8b778bacf758448385f5d3c7dbe502f7",
      "b27bcf76e53f4035ad0464bcf0fe490f",
      "16c60bd460c949bc97c9820c2c0d971e",
      "d47976209d6f49d2ac596fda9e926393",
      "3ddaced9ad7b4ad3b9fc9629e8d0e240",
      "c8394ae833884a4ab89622c2597dd1ae",
      "ed76b36c51f443ca829c6538eba769d0",
      "4aa1b456d232482c8b8ae0f090c8eee4",
      "286792edbe7b412e95a49912378213b8",
      "4435c65b37fd483ca6fd6df94381f010",
      "a6bbcc9b57094e988c4c44eec6a720f4",
      "fb70739a2c8f40ca9a992851a47c64ec",
      "3312fae1e90c42d4929c95e2b90e669b",
      "ca82a302c2a14c9aaea2ef3a8185777f",
      "1a51eb4e60bd495b8f457533b43032e4",
      "80d299714ef04cd9801338d64af029f9"
     ]
    },
    "id": "IxeJBA6Y3pUt",
    "outputId": "e2995d0c-0889-41bc-a20c-27c9275c3cf0"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3a893b9431744117ac9b7e50c2a9f013",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "df272f940b3341bdb5eb15b8084b11af",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8394ae833884a4ab89622c2597dd1ae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 4.4407859e+00 -5.9886807e-01 -1.7820705e+00 ... -6.0590094e-01\n",
      "   -2.4691533e-01 -9.4437933e-01]\n",
      "  [ 8.8075218e+00 -2.2492611e+00 -2.0816329e+00 ... -1.6342618e+00\n",
      "   -1.0839297e+00 -1.6352931e+00]\n",
      "  [ 8.7270842e+00 -2.3283825e+00 -2.1411471e+00 ... -1.4970571e+00\n",
      "   -1.0620427e+00 -1.4405980e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.2898493e+00 -1.2386279e+00 -2.1868134e+00 ... -1.0399776e+00\n",
      "    1.6760746e-01  5.0261843e-01]\n",
      "  [ 8.7246866e+00 -2.1077049e+00 -2.2196045e+00 ... -1.5790584e+00\n",
      "   -1.1902786e+00 -1.6472648e+00]\n",
      "  [ 9.0957546e+00 -2.4205801e+00 -2.1287379e+00 ... -1.3080864e+00\n",
      "   -1.7220010e+00 -1.3419683e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 5.7606506e-01 -2.1305047e-01 -7.2901160e-01 ...  2.5504664e-01\n",
      "   -6.6497529e-01  2.1282965e-01]\n",
      "  [-1.5775198e+00  1.8525468e+00 -2.7846406e+00 ... -8.8204771e-01\n",
      "    7.2834539e-01 -3.4666426e+00]\n",
      "  [-1.7270980e+00 -2.2520604e+00  1.7191662e+00 ...  5.5982461e+00\n",
      "   -2.4151659e+00  5.2771986e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[-3.3774820e-01  8.4328771e-01  1.1345463e+00 ... -1.1094406e-01\n",
      "   -1.1364087e+00 -2.4013951e-01]\n",
      "  [ 6.5644140e+00 -8.8308448e-01 -3.3068452e+00 ... -1.9205462e+00\n",
      "   -5.7633102e-01 -2.2466795e+00]\n",
      "  [ 6.4640865e+00 -1.4332345e+00 -2.7109480e+00 ... -1.6428434e+00\n",
      "   -1.5020180e+00 -2.0159752e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 2.2832419e-01 -2.3189896e-01 -5.8768260e-01 ...  1.0473437e+00\n",
      "   -1.0542186e+00  6.6157177e-02]\n",
      "  [-1.6325519e+00  1.4676071e+00 -3.1423402e+00 ... -7.9037172e-01\n",
      "    5.0300092e-01 -3.4364984e+00]\n",
      "  [-1.3904735e+00 -1.5975631e+00 -1.0392002e+00 ...  3.8778534e+00\n",
      "   -1.5898582e+00 -1.2503107e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.2335207e-01 -1.9086435e-01 -5.8925468e-01 ...  3.3766517e-01\n",
      "   -7.3954445e-01  2.9016656e-01]\n",
      "  [-2.4388726e+00  2.4658167e+00 -3.0103219e+00 ... -1.5446794e+00\n",
      "    2.1134079e+00 -3.3421557e+00]\n",
      "  [-2.3842170e+00 -2.7188706e+00  2.7929008e+00 ...  5.2994766e+00\n",
      "   -2.6849985e+00  2.2444813e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n",
      "F1-score of [de] model on [it] dataset: 0.674\n"
     ]
    }
   ],
   "source": [
    "\n",
    "f1_scores[\"de\"][\"it\"] = evaluate_lang_performance(\"it\", trainer)\n",
    "print(f\"F1-score of [de] model on [it] dataset: {f1_scores['de']['it']:.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "referenced_widgets": [
      "a6f991c86951454f853978e08d81f36c",
      "5c87ecea200e43e5b757469cf8a62d3c",
      "5df28d2528834a66a1e9e4962dd8b31f",
      "3b596911fa7a4821b62de803328477d7",
      "ba197b6682044734b5a4ccd92a78c945",
      "9c1534cf995e47a5a31f1ede6f4e2926",
      "74e18ecf7cea46b18feab1704c01b6cb",
      "913b9d3327f24e2fb3f63ca8fc975572",
      "134a5f727b8545d0b2442f77ba19ba85",
      "5535ea8576404174b17cb4773c14112e",
      "4a94340d8b084cc6be922f77465a64e9",
      "bdf37dd464d24f8da6212c560fcc4ddc",
      "276fb4f0480046b69b296d034af6a11f",
      "fa21c19946574d4db8be3a94708685a7",
      "c4558d4453124b539d6dae2d77e0cc22",
      "6a3ebada114f49879e650cb890e713ea",
      "f7094e82152a4f988e9a51612ddf57a3",
      "b9b5e45f30fc48778aa6f2e0298652e4",
      "3fffcbd9bd684b8bbcb58d127b7b98ca",
      "4fa2f7d96558420eaadfc8c9bbdd0119",
      "7db222dc3fb9457bae8daf4937977699",
      "6685a46a80f046d7a7108dc3812573f4",
      "93de5543aba84405a952dee2aa257508",
      "a468093f9adf410596d7ca2dd880f05c",
      "2128c0ba61d8423a9d92503defbed462",
      "df2eecd3afb743d3bdbe58b8a1c268a9",
      "3d5649e2ee0549c1a1a576b47bb2b2ad",
      "c747523f9e0b473a9cacbd10ca7d5d16",
      "8cd59a5a34d148d59b2c8fef23ac69b1",
      "a29fbee89c4b4289a3323d4b1fda6293",
      "56596bff7af94bb8b78e027b5d347e3d",
      "f3e27114c3ef4c3c94dda99c0c04ef9f",
      "a7fef51748b7414796602bf7d2bc3ed6"
     ]
    },
    "id": "JsF12IyN3pUu",
    "outputId": "fe832c29-ec62-4197-d5c0-113b1c5e8476"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a6f991c86951454f853978e08d81f36c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bdf37dd464d24f8da6212c560fcc4ddc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "93de5543aba84405a952dee2aa257508",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 4.7665224e+00 -9.3068606e-01 -2.1274714e+00 ... -4.7810385e-01\n",
      "   -6.6833413e-01 -9.6580273e-01]\n",
      "  [ 8.7561617e+00 -2.4981470e+00 -1.7115521e+00 ... -8.0573434e-01\n",
      "   -1.8492281e+00 -1.3500013e+00]\n",
      "  [ 8.9736013e+00 -2.7768168e+00 -1.9131891e+00 ... -9.6938354e-01\n",
      "   -1.7436916e+00 -1.1427679e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.9479184e-01  1.7378517e+00  1.5029880e+00 ... -8.9298761e-01\n",
      "   -1.4851245e+00 -1.0494688e+00]\n",
      "  [-1.8140838e+00  7.8373380e+00 -7.5080079e-01 ... -1.8007071e+00\n",
      "   -1.2744637e+00 -2.9514327e+00]\n",
      "  [-2.0337076e+00 -5.3698683e-01  7.4742999e+00 ...  7.8361773e-01\n",
      "   -2.4127264e+00 -7.7988231e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.0456167e-01 -1.6979107e-01 -6.9596964e-01 ...  1.4598557e-01\n",
      "   -5.9311980e-01  3.0747527e-01]\n",
      "  [-1.9020087e+00  2.7116208e+00 -3.6081233e+00 ... -1.8935308e+00\n",
      "    3.2223771e+00 -3.3840294e+00]\n",
      "  [-3.0828643e+00 -1.6364251e+00  7.6853949e-01 ...  3.0426805e+00\n",
      "   -3.9957848e-01  1.9520038e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[-1.0130343e+00  2.0378072e+00  2.2203360e+00 ... -4.0021047e-01\n",
      "   -1.0428988e+00 -1.0306203e+00]\n",
      "  [-2.0641263e+00  7.7818623e+00 -6.1204529e-01 ... -1.8068923e+00\n",
      "   -8.6191970e-01 -2.8961327e+00]\n",
      "  [-1.9946129e+00 -4.3732053e-01  7.5021896e+00 ...  7.3149437e-01\n",
      "   -2.4040153e+00 -6.0447299e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.1186981e+00 -3.7030503e-01 -1.0185798e+00 ... -1.6772585e-01\n",
      "   -4.3479624e-01  3.4612525e-01]\n",
      "  [ 8.4524755e+00 -1.6578344e+00 -1.9185865e+00 ... -1.3113390e+00\n",
      "   -1.7197523e+00 -2.2595742e+00]\n",
      "  [ 8.5714273e+00 -1.7101010e+00 -2.0553975e+00 ... -1.2579901e+00\n",
      "   -1.6771709e+00 -2.1002901e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.3509963e+00 -7.0866197e-01 -1.3457195e+00 ...  6.0917240e-01\n",
      "   -8.2373077e-01  1.5949946e-02]\n",
      "  [ 3.4660134e-01  2.2885075e-01 -3.0582433e+00 ... -7.7538413e-01\n",
      "    1.1857339e+00 -3.2914548e+00]\n",
      "  [ 2.1867669e+00 -3.2369406e+00 -4.2533083e-03 ...  3.7264447e+00\n",
      "   -2.3028104e+00  1.2075316e-02]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n",
      "F1-score of [de] model on [en] dataset: 0.592\n"
     ]
    }
   ],
   "source": [
    "\n",
    "f1_scores[\"de\"][\"en\"] = evaluate_lang_performance(\"en\", trainer)\n",
    "print(f\"F1-score of [de] model on [en] dataset: {f1_scores['de']['en']:.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "l_nbRoQd3pUu",
    "outputId": "8a45a7d9-aa63-4c2d-87b2-da5424cc8ac7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1-score of [de] model on [en] dataset: 0.592\n"
     ]
    }
   ],
   "source": [
    "#hide_input\n",
    "print(f\"F1-score of [de] model on [en] dataset: {f1_scores['de']['en']:.3f}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## When Does Zero-Shot Transfer Make Sense?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "id": "RLZs5r6m3pUu"
   },
   "outputs": [],
   "source": [
    "def fine_tuning_training_on_single_corpus(dataset, num_samples):\n",
    "    \"\"\"\n",
    "    Function to train the model on a single corpus of data.\n",
    "\n",
    "    Parameters:\n",
    "    dataset (DatasetDict): The dataset to train on. It should be a HuggingFace DatasetDict containing 'train', 'validation' and 'test' splits.\n",
    "    num_samples (int): The number of samples from the training set to use for training.\n",
    "\n",
    "    Returns:\n",
    "    results (pd.DataFrame): A pandas DataFrame containing the number of training samples used and the F1 score on the test set.\n",
    "    \"\"\"\n",
    "    # Shuffle the training data and select the first 'num_samples' examples.\n",
    "    train_ds = dataset[\"train\"].shuffle(seed=42).select(range(num_samples))\n",
    "    # The validation and test sets are not shuffled or truncated.\n",
    "    valid_ds = dataset[\"validation\"]\n",
    "    test_ds = dataset[\"test\"]\n",
    "\n",
    "    # Update the logging steps in the training arguments to log progress after each batch.\n",
    "    training_args.logging_steps = len(train_ds) // batch_size\n",
    "\n",
    "    # Initialize a Trainer instance. This is a HuggingFace class that handles training.\n",
    "    trainer = Trainer(model_init=model_init, args=training_args,\n",
    "        data_collator=data_collator, compute_metrics=compute_metrics,\n",
    "        train_dataset=train_ds, eval_dataset=valid_ds, tokenizer=xlmr_tokenizer)\n",
    "    \n",
    "    # Train the model.\n",
    "    trainer.train()\n",
    "\n",
    "    # If the training arguments specify to push the model to the HuggingFace model hub, do so with a commit message.\n",
    "    if training_args.push_to_hub:\n",
    "        trainer.push_to_hub(commit_message=\"Training completed!\")\n",
    "    \n",
    "    # After training, compute the F1 score on the test set.\n",
    "    f1_score = get_f1_score(trainer, test_ds)\n",
    "\n",
    "    # Return the results as a pandas DataFrame.\n",
    "    return pd.DataFrame.from_dict(\n",
    "        {\"num_samples\": [len(train_ds)], \"f1_score\": [f1_score]})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 104,
     "referenced_widgets": [
      "84759f2e60c14548b285c0b32c39230b",
      "8a546e75a87245309f7a56eef8a68d9a",
      "32223b0d3b624f78acbfbfb97f6917db",
      "38c0b45e3ab448df8d8eef15a703d26c",
      "6913c066eefa4d769d556d9c6403b634",
      "4ea3cd5b83024857a882f59b4a2ffe92",
      "67f229dbd1e449069eec44e0be7bf048",
      "de214be9ed654a5f83a9bd083b4deba6",
      "d69991d3c44f41ef988090bddec7bbc4",
      "d2f04c99f8384f9ebeb486b668c2881b",
      "b52c73cb5a034b388682d92ada5ec67e"
     ]
    },
    "id": "rY8cjHhr3pUv",
    "outputId": "94c37dcf-9412-4a77-dbf1-0fc2c0fa5046"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "84759f2e60c14548b285c0b32c39230b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.fr/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-df08c9bf8a078f41.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.fr/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-5a8067996ffe172e.arrow\n"
     ]
    }
   ],
   "source": [
    "\n",
    "panx_fr_encoded = encode_panx_dataset(panx_ds_combined[\"fr\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "rAteKPwy3pUv",
    "outputId": "1de9069f-c7fd-441e-e7c0-902c0fd0904b"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:310: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  FutureWarning,\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='33' max='33' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [33/33 00:23, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1.549200</td>\n",
       "      <td>1.302521</td>\n",
       "      <td>0.023641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1.163700</td>\n",
       "      <td>1.039713</td>\n",
       "      <td>0.202990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.978600</td>\n",
       "      <td>0.957213</td>\n",
       "      <td>0.257165</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[-1.8655172e-01  2.5774515e-01 -6.4279777e-01 ...  2.8908756e-01\n",
      "   -2.9562530e-01  4.6173934e-02]\n",
      "  [ 4.2187893e-01 -3.4984612e-01 -6.8802446e-01 ...  1.0128658e-01\n",
      "   -4.6693420e-01  1.0614473e-01]\n",
      "  [ 4.7852001e-01 -3.8667294e-01 -5.5348992e-01 ...  2.9050598e-01\n",
      "   -6.1689746e-01  1.1553648e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.8378653e-01 -1.7224382e-01 -8.0492139e-01 ...  1.8132584e-02\n",
      "   -4.7344595e-01  3.7336785e-01]\n",
      "  [ 1.5976342e+00 -7.5798559e-01 -8.3334017e-01 ...  5.2848744e-01\n",
      "   -3.3168402e-01 -5.1320523e-01]\n",
      "  [ 1.0831923e+00 -8.4746349e-01 -7.2670305e-01 ...  2.5918084e-01\n",
      "   -4.9027973e-01 -1.9299954e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.4720010e-01 -1.7289527e-02 -8.2395649e-01 ...  1.1141926e-01\n",
      "   -4.4698009e-01  2.7766311e-01]\n",
      "  [ 4.3363295e+00 -5.4112846e-01 -6.0440880e-01 ... -9.5253453e-02\n",
      "   -6.1437488e-01 -1.2796431e+00]\n",
      "  [ 4.0010147e+00 -3.6103871e-01 -1.6358347e-01 ...  3.1851426e-02\n",
      "   -3.5779709e-01 -1.5047817e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 1.8881778e-01 -2.4968127e-02 -7.7343315e-01 ...  9.6113466e-02\n",
      "   -5.3023434e-01  2.7496347e-01]\n",
      "  [ 3.1522615e+00 -8.3278579e-01 -9.0052992e-01 ...  1.9481528e-01\n",
      "   -7.3408300e-01 -1.1405092e+00]\n",
      "  [ 3.4188700e+00 -8.8745403e-01 -9.1205068e-02 ...  3.4165585e-01\n",
      "   -5.7458353e-01 -1.4989884e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.3547659e-01 -4.0120620e-02 -7.8793788e-01 ...  1.3668670e-01\n",
      "   -3.7452000e-01  2.5451890e-01]\n",
      "  [ 1.6679837e+00 -1.0250416e+00 -7.8529835e-01 ...  7.8839689e-01\n",
      "    4.1143898e-02 -6.6373885e-01]\n",
      "  [ 8.7112260e-01 -8.7109685e-01 -6.8014228e-01 ...  2.9040006e-01\n",
      "   -5.0489658e-01 -1.0938579e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.5826182e-01 -6.7664482e-02 -7.8997451e-01 ...  7.7952042e-02\n",
      "   -4.8284781e-01  3.5080302e-01]\n",
      "  [ 7.2508156e-01 -5.3766453e-01 -8.9778197e-01 ...  2.8652146e-01\n",
      "   -3.7180912e-01  8.3751697e-03]\n",
      "  [ 7.4347979e-01 -6.2499374e-01 -6.2626159e-01 ...  5.2743393e-01\n",
      "   -5.9450716e-01  1.1000854e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n",
      "eval_pred.predictions  [[[-2.94059634e-01  2.28750989e-01 -6.97800338e-01 ...  2.23695755e-01\n",
      "   -3.66611481e-01  2.65927255e-01]\n",
      "  [-1.25916496e-01 -3.54742676e-01 -5.81280828e-01 ...  1.84962004e-01\n",
      "   -2.30513588e-01  1.49240494e-01]\n",
      "  [-4.76770811e-02 -3.24829489e-01 -4.98474777e-01 ...  4.10861671e-01\n",
      "   -4.25468773e-01  2.35122889e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[-2.16516927e-01 -8.77394713e-03 -7.25851655e-01 ...  2.28007078e-01\n",
      "   -4.33219075e-01  3.83845985e-01]\n",
      "  [ 3.08080435e+00 -1.29999328e+00 -8.62935483e-01 ...  1.90631360e-01\n",
      "    8.04005712e-02 -1.02211046e+00]\n",
      "  [ 1.94518375e+00 -1.34121442e+00 -3.98955405e-01 ...  8.98667425e-02\n",
      "   -1.75600857e-01 -6.65713787e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 1.17865138e-01  2.22344801e-01 -7.99595535e-01 ...  2.28157803e-01\n",
      "   -3.93233895e-01  2.33565718e-01]\n",
      "  [ 4.35050058e+00 -9.30084050e-01 -1.09650385e+00 ... -5.88145316e-01\n",
      "   -1.09682667e+00 -6.33806825e-01]\n",
      "  [ 4.38030195e+00 -9.03893471e-01 -1.04893744e+00 ... -4.29020524e-01\n",
      "   -1.06187630e+00 -6.88617468e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[-7.68279210e-02 -1.92454569e-02 -7.63898075e-01 ...  1.39956936e-01\n",
      "   -5.27824759e-01  2.98832119e-01]\n",
      "  [ 4.15967560e+00 -1.31385994e+00 -1.10334766e+00 ... -3.27105790e-01\n",
      "   -1.16846967e+00 -1.12588358e+00]\n",
      "  [ 4.08402634e+00 -1.35878909e+00 -5.78904331e-01 ... -1.67350814e-01\n",
      "   -1.12715721e+00 -1.08740485e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[-4.35295314e-01  1.01175480e-01 -7.19838142e-01 ...  2.21451834e-01\n",
      "   -2.57302910e-01  3.19653749e-01]\n",
      "  [ 2.91360688e+00 -1.23905909e+00 -9.12690580e-01 ...  2.76040971e-01\n",
      "    1.99840307e-01 -8.95979285e-01]\n",
      "  [ 1.71389782e+00 -1.46123075e+00 -5.70750296e-01 ...  2.81784475e-01\n",
      "   -2.45537803e-01 -4.95546699e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 1.20187454e-01  4.98654246e-02 -8.16760421e-01 ...  1.93541318e-01\n",
      "   -4.28392261e-01  3.73455167e-01]\n",
      "  [-1.83733538e-01 -7.64217317e-01 -7.83748925e-01 ...  8.24053347e-01\n",
      "    2.76656270e-01 -4.57955420e-01]\n",
      "  [-5.54461777e-01 -6.40485525e-01 -2.32732102e-01 ...  1.33269083e+00\n",
      "   -2.04090215e-02 -1.33248582e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]]\n",
      "eval_pred.predictions  [[[-8.1877060e-02  1.3356450e-01 -7.3778296e-01 ...  6.3314877e-02\n",
      "   -4.6138746e-01  3.7106621e-01]\n",
      "  [-2.8752023e-01 -2.6461038e-01 -5.2557111e-01 ...  1.0159917e-01\n",
      "   -1.9023417e-01  2.2473356e-01]\n",
      "  [-1.7223762e-01 -2.3193265e-01 -4.1895935e-01 ...  3.2736099e-01\n",
      "   -4.4001931e-01  3.7950489e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[-1.2900625e-01 -3.5899054e-02 -7.3808104e-01 ...  1.2820747e-01\n",
      "   -4.7737324e-01  4.8851258e-01]\n",
      "  [ 3.4608750e+00 -1.1679413e+00 -1.1247991e+00 ... -2.2474064e-01\n",
      "   -3.0922621e-01 -5.0786787e-01]\n",
      "  [ 2.8095992e+00 -1.3554330e+00 -6.5535271e-01 ... -3.0296460e-01\n",
      "   -3.5894662e-01 -4.4716635e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 2.3206793e-01  1.8327616e-01 -8.0509841e-01 ...  1.0852701e-01\n",
      "   -4.6038297e-01  2.9717350e-01]\n",
      "  [ 4.8447661e+00 -9.6577924e-01 -1.0655619e+00 ... -9.3117148e-01\n",
      "   -1.2267649e+00 -7.4654812e-01]\n",
      "  [ 4.9230843e+00 -9.4967914e-01 -1.0048610e+00 ... -7.6687372e-01\n",
      "   -1.1790489e+00 -8.0246633e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 1.3636722e-01 -5.3872839e-02 -8.1280994e-01 ...  5.3427629e-02\n",
      "   -5.8322012e-01  3.8181520e-01]\n",
      "  [ 4.5674820e+00 -1.2976944e+00 -1.0788457e+00 ... -6.5889436e-01\n",
      "   -1.3413161e+00 -1.0850446e+00]\n",
      "  [ 4.3520751e+00 -1.3357915e+00 -4.7881219e-01 ... -4.9268150e-01\n",
      "   -1.2594510e+00 -1.0077388e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[-3.3297535e-02 -4.7514495e-02 -7.9422927e-01 ...  3.5692297e-02\n",
      "   -4.0378284e-01  4.9002078e-01]\n",
      "  [ 3.2730587e+00 -1.0634594e+00 -1.0555152e+00 ... -1.9178659e-01\n",
      "   -1.5428370e-01 -5.5406201e-01]\n",
      "  [ 2.4716704e+00 -1.4039518e+00 -6.4167792e-01 ... -1.3741203e-01\n",
      "   -4.3905780e-01 -3.4389374e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.5433003e-01  3.4589007e-02 -8.2564634e-01 ...  1.4134502e-01\n",
      "   -4.5933795e-01  4.0929884e-01]\n",
      "  [-5.3715652e-01 -6.3018513e-01 -5.7244420e-01 ...  6.6327387e-01\n",
      "    4.3858048e-01 -4.5725769e-01]\n",
      "  [-8.9154238e-01 -5.6600451e-01 -1.9327750e-02 ...  1.1773189e+00\n",
      "    1.7152382e-01 -7.2431281e-02]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 3.87685716e-01 -3.97074334e-02 -7.73876488e-01 ...  1.73948500e-02\n",
      "   -5.21487713e-01  3.63628745e-01]\n",
      "  [ 4.70920897e+00 -9.36288059e-01 -1.12213516e+00 ... -1.01804352e+00\n",
      "   -1.29782093e+00 -8.55569541e-01]\n",
      "  [ 4.79363060e+00 -9.74861145e-01 -9.88769770e-01 ... -8.08017671e-01\n",
      "   -1.33430743e+00 -8.09566796e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 1.11038469e-01  1.03455134e-01 -7.53915191e-01 ...  1.42089397e-01\n",
      "   -4.63007361e-01  4.04197067e-01]\n",
      "  [ 3.51252031e+00 -6.27119660e-01 -6.83519781e-01 ... -1.58322960e-01\n",
      "   -4.25105274e-01 -1.25549531e+00]\n",
      "  [ 3.26833558e+00 -9.88096714e-01 -2.52010822e-01 ...  1.38338298e-01\n",
      "   -5.82218051e-01 -1.37100863e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 3.46014172e-01  2.13670321e-02 -7.33035624e-01 ... -8.27634893e-03\n",
      "   -4.75329667e-01  3.23311299e-01]\n",
      "  [ 4.18925714e+00 -1.78628111e+00 -8.47095475e-02 ... -4.11054701e-01\n",
      "   -5.58629155e-01 -1.32072973e+00]\n",
      "  [ 1.10027218e+00 -7.43160307e-01 -4.10654575e-01 ... -1.16740212e-01\n",
      "   -2.64700294e-01 -3.37539136e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[-7.67992213e-02  1.93623215e-01 -8.25779617e-01 ...  2.38333434e-01\n",
      "   -4.88755405e-01  3.97929996e-01]\n",
      "  [ 4.43984413e+00 -1.06357336e+00 -1.14256346e+00 ... -5.66120625e-01\n",
      "   -1.33998096e+00 -7.06958890e-01]\n",
      "  [-1.19595301e+00 -2.73694843e-01  1.19464852e-01 ...  1.05118692e+00\n",
      "    3.89204741e-01 -4.66583610e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 7.48118535e-02 -5.49823381e-02 -7.89234221e-01 ...  7.07405433e-02\n",
      "   -4.57853824e-01  4.85354394e-01]\n",
      "  [-2.90073156e+00 -1.24808677e-01  1.35180697e-01 ...  8.53669465e-01\n",
      "    8.55457127e-01  1.94017619e-01]\n",
      "  [-2.26835799e+00 -2.27996022e-01  2.19057381e-01 ...  1.09304476e+00\n",
      "    8.00428987e-01  3.66331302e-02]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 3.39015394e-01  6.44628378e-03 -7.92722285e-01 ...  5.92994690e-02\n",
      "   -4.72633749e-01  3.05792212e-01]\n",
      "  [ 4.79972172e+00 -9.68887925e-01 -1.07143664e+00 ... -1.02155375e+00\n",
      "   -1.31971681e+00 -8.87373924e-01]\n",
      "  [ 4.84379911e+00 -9.47343051e-01 -1.00126183e+00 ... -8.50666761e-01\n",
      "   -1.37529624e+00 -9.12594497e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-1d547b6c-00a8-45f1-b549-eb11fc041155\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>num_samples</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>250</td>\n",
       "      <td>0.257339</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1d547b6c-00a8-45f1-b549-eb11fc041155')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-1d547b6c-00a8-45f1-b549-eb11fc041155 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-1d547b6c-00a8-45f1-b549-eb11fc041155');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "   num_samples  f1_score\n",
       "0          250  0.257339"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "# training_args.push_to_hub = False\n",
    "metrics_df = fine_tuning_training_on_single_corpus(panx_fr_encoded, 250)\n",
    "metrics_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81
    },
    "id": "TbcWlppp3pUv",
    "outputId": "85c5b889-5829-4e2b-b4cc-899a7763803d"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-f6b90776-632f-4983-af7f-f959a64d248d\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>num_samples</th>\n",
       "      <th>f1_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>250</td>\n",
       "      <td>0.257339</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f6b90776-632f-4983-af7f-f959a64d248d')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-f6b90776-632f-4983-af7f-f959a64d248d button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-f6b90776-632f-4983-af7f-f959a64d248d');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "   num_samples  f1_score\n",
       "0          250  0.257339"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#hide_input\n",
    "# Hack needed to exclude the progress bars in the above cell\n",
    "metrics_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "zxCdayoV3pUw",
    "outputId": "fb553f82-aaf4-4ba5-c480-f536a8fb14d6"
   },
   "outputs": [],
   "source": [
    "\n",
    "for num_samples in [500, 1000, 2000, 4000]:\n",
    "    metrics_df = metrics_df.append(\n",
    "        fine_tuning_training_on_single_corpus(panx_fr_encoded, num_samples), ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 283
    },
    "id": "DpYWn_073pUw",
    "outputId": "48b89b74-e0ea-4097-aaae-5cc1d282b578"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXwV5dn/8c+VhCQsERRQ2UGroCwGDVRrUbRVcQO1Vos+rVtLhbo9Lj+1Fkvt6tZiLaVVa6lLK25t0WK18oB7awKibLJpKpuCEJBIErJcvz9mEk6SkxBCTs4h832/XueV2ec6k+S+Zu6ZuW9zd0REJLrSkh2AiIgklxKBiEjEKRGIiEScEoGISMQpEYiIRJwSgYhIxCUsEZjZw2a20cwWNzDfzOzXZrbKzN4zs6MTFYuIiDQskVcEM4Axjcw/HTgs/EwApicwFhERaUDCEoG7vwpsaWSRccAjHvg30MXMeiQqHhERiS8jifvuBayJGV8bTttQd0Ezm0Bw1UDHjh2PGTRoUKsEKCLSVsyfP/9Td+8eb14yE0GTufsDwAMAeXl5XlBQkOSIRET2LWb234bmJfOpoXVAn5jx3uE0ERFpRclMBLOAb4VPDx0LbHP3etVCIiKSWAmrGjKzvwCjgW5mthb4IdAOwN1/B8wGzgBWATuAyxIVi4iINCxhicDdx+9mvgPfS9T+RUSkafRmsYhIxCkRiIhEnBKBiEjEKRGIiEScEoGISMQpEYiIRJwSgYhIxCkRiIhEnBKBiEjEKRGIiEScEoGISMQpEYiIRJwSgYhIxCkRiIhEnBKBiEjEKRGIiEScEoGISMQpEYiIRJwSgYhIxCkRiIhEnBKBiEjEKRGIiEScEoGISMQpEYiIRJwSgYhIxCkRiIhEnBKBiEjEZSQ7ABERCbg7peVVFJdVsGNnRfizks/LKvi8rJLBPfejf7eOLb5fJQIRkWYqr6xiR1kln++sCArrmkK7IpxWWWv6jlrTwuFw3ertVHnD+/vxOUOUCEREmsvdKSmvDM6yyyprn21XF+RlleGZeOWuM/LYgr5Oob+zoqrJ+++YmU7HrIzwk06HzAy6dcqkX1YHOmbumt4xK6Nm2Q6ZGXTKyqBDVjqdsjI4aL/shBwbJQIRSUk7K6pqVY/UK5Srz7LL6hTc1cvGnH3v2Bms542cbcfKTE+rKaxjC+LuOVlhQZ1Rq8CutWydAr9jZgbt26WTlmaJPWB7QYlARPZaVZWzo7ySHU2oHomtCimOKaRjz7h3lFWys7JpZ9tmhAVzek0B3SEznYNysunQLaawrj7LzsqgU52Cu1P19MwM2memk5kRredolAhEIqisorJWQbzrDDq2+mNXwb2jrJLincHZd93qkeoz7qbKzEirXQCHPw/KyY5bPdIxMzgj33Umnl6rKqV9u3TMUvdse1+gRCCS4qqqPCiMd9at367/VEm86pHisvBMPSy4d+ysoLyyaXUkaTVn27uqRzpkptOjczYdYgrlRs+yw+nVZ+rt0qN1tr0vUCIQaUHuTllFVc1Zct266rrVI7F12LWWjVmnpLzpZ9vZ7dJ2nUGHhW/n9u3o2Tm7Tp12MBwU1DH12pmxZ+QZZLdL09l2BCQ0EZjZGOA+IB14yN1/UWd+X+BPQJdwmVvcfXYiYxLZnYrKKj7ZXsaGrSWs21rCpu1lFMeeUcfcnKz/yGAllY09/xcjzaBjVp0z6MwMenbJrFU9UveGZYc41SMdszLo0C6dDJ1tSzMkLBGYWTowDTgFWAvkm9ksd18as9gPgCfdfbqZHQnMBvonKiYRd6doRznrt5bUfDZsK2Vd+HP91hI++aw07rPc7dul7yp0M4NCukuHTHrtX7tQbrR6JOaMOytDZ9uSGhJ5RTASWOXuHwCY2RPAOCA2ETiwXzjcGVifwHgkAkp2VrJ+W0lMQV9aU9iv31rC+m0llJbXfholMyONnp2z6dG5PV86tBs9u2TTs0t7enTOpleX9hy4XzadsjJIT+HH/0T2RiITQS9gTcz4WuCLdZaZArxkZlcDHYGvxtuQmU0AJgD07du3xQOVfUNFZRUbt5eFBXppnMK+hKId5bXWMYMDc7Lo0bk9R/TYj5MHHUjPLu1rCvueXdrTtWOmzswl0pJ9s3g8MMPd7zWz44BHzWyIu9c6ZXP3B4AHAPLy8pr4SojsS9ydrTvKa1XRxBb4G7aW8HGcKpuc7Ax6hQX68L5ddhXynYNpB+2XHblnwkX2VCITwTqgT8x473BarCuAMQDu/paZZQPdgI0JjEuSoLrKZkN49r4uPINfv7W0piqnXpVNeho9wkL92EO70qtLe3p0bl+r6iYnu12SvpFI25HIRJAPHGZmAwgSwDeAi+os8xHwFWCGmR0BZAObEhiTJEB1lc2GbSWs21rKhvBsft3W0rCwj19l071TFj26tGfQwTmcPPBAenRpT68u2WFhH1TZpPJr+SJtRcISgbtXmNlVwIsEj4Y+7O5LzOwOoMDdZwE3AA+a2f8S3Di+1L2prYFIa6iuslkfnr0HhX04HBb4n2wvq/fIZE52Rlg9k01un11VNj06t6eXqmxEUorta+VuXl6eFxQUJDuMNqO0vHLXDdfw7H1DOLwuHK77QlNmehoHd87edcM1PIPv0SU7rL5RlY1IqjGz+e6eF29esm8WSwK5Ox9/VlrvMcp1Mc/Pb/l8Z731uudk0bNLewYelMNJA8OnbDpn1xT23TpmqcpGpA1RImijFq7Zyg9nLeHdNVtrTc/Jyqgp0I/q06XmDL76zP6gzllkZaQnKWoRSQYlgjZm0/Yy7vrn+zw1fy0H5mTxgzOP4NADO9Gzc1D476cqGxGpQ4mgjSivrOJPbxZy38srKa2o5LsnHsLVJx9Gpyz9ikWkcSol2oDXV37KlOeWsGpjMSce3p3bzz6SQ7t3SnZYIrKPUCLYh63ZsoOf/mMZ/1zyMX0P6MBD38rjK0ccqOYSRGSPKBHsg0p2VjL9ldX8/pXVpJlx02kDueLLA8hup5u8IrLnlAj2Ie7OPxd/zE/+sYx1W0s4+6iefP+MQfTo3D7ZoYnIPkyJYB+x4pPt/Oi5JbyxajODDs7hiQnHcuwhXZMdloi0AUoEKW5bSTlTX17BI2/9l05ZGdwxbjAXjeyrnqhEpMUoEaSoqirnqflruOufy9myYyfjR/blxlMHckDHzGSHJiJtjBJBCnrnoyKmzFrCu2u3kddvf/40diRDenVOdlgi0kYpEaSQTdvLuPOf7/N0+Fbwry48inNye+lxUBFJKCWCFKC3gkUkmfa9kmb5chg9uva0Cy6ASZNgxw4444z661x6afD59FM4//z68ydOhAsvhDVr4JvfrD//hhvg7LODfX/3u/Xn/+AH8NWvwsKFcN119ef/7GfwpS/Bm2/C979fa9Zrnfvxoy9exKrPKhjdNY3b5zzIIW8UwZ0xC/3+9zBwIDz3HNx7b/3tP/oo9OkDM2fC9On15z/9NHTrBjNmBJ+6Zs+GDh3gt7+FJ5+sP3/evODnPffA88/Xnte+PbzwQjD84x/DnDm153ftCs88Ewzfeiu89Vbt+b17w2OPBcPXXRccw1iHHw4PPBAMT5gAK1bUnp+bC1OnBsP/8z+wdm3t+ccdBz//eTD8ta/B5s2153/lKzB5cjB8+ulQUlJ7/llnwY03BsN1/+5gn/7bA4Jjl5sLL78MP/lJ/fn62wuG2+LfXox9LxG0EWuyOvOTfqN58YDD6Vfl/OGSPE5e+x72j6JkhyYiEaOOaVpZ3beCrzr5C3orWEQSTh3TpAB354XFH/NTvRUsIilGiaAVrPhkO1NmLeHN1cFbwTMnHMsX9VawiKQIJYIE2lZSzq/+tYJH/x28FfzjcYMZr7eCRSTFKBEkQFWV82TBGu56cTlFeitYRFKcEkELWxC+Ffxe+FbwFL0VLCIpTomghWzcXsqdLyznmQXBW8FTL8xlXG5PvRUsIilPiWAv7awI3wqes5KyikquPPFQrjr5C3orWET2GSqt9sJrKzcxZdYSVm/6nNEDu3P7WUdyiPoKFpF9jBJBM63eVMwlD79NnwM6BG8FD1JfwSKyb1IiaKY3V2+myuGRy0fSr2vHZIcjItJseqC9mQoKt3BgThZ9D+iQ7FBERPaKEkEzFRQWMWLAAaoOEpF9nhJBM6zbWsK6rSWM6Ld/skMREdlrSgTNUFC4BYC8/gckORIRkb2nRNAM+YVb6JSVwaCDc5IdiojIXlMiaIaCwiKG9+2ixuNEpE1QSbaHtu0oZ/kn2xmpaiERaSMSmgjMbIyZLTezVWZ2SwPLXGBmS81siZn9OZHxtIT5H23BXfcHRKTtSNgLZWaWDkwDTgHWAvlmNsvdl8YscxhwK3C8uxeZ2YGJiqel5BcWkZFm5PbpkuxQRERaRCKvCEYCq9z9A3ffCTwBjKuzzHeAae5eBODuGxMYT4soKNzCkF6daZ+pPoZFpG1IZCLoBayJGV8bTot1OHC4mb1hZv82szHxNmRmE8yswMwKNm3alKBwd6+0vJJ312xjRH+9PyAibUeybxZnAIcBo4HxwINmVq/Oxd0fcPc8d8/r3r17K4e4y+J129hZWcUI3R8QkTakSYnAzL5sZpeFw93NbEATVlsH9IkZ7x1Oi7UWmOXu5e7+IbCCIDGkpLfDF8mO0RvFItKG7DYRmNkPgZsJbuoCtAMea8K284HDzGyAmWUC3wBm1VnmbwRXA5hZN4Kqog+aFHkSFBQWcWj3jnTtlJXsUEREWkxTrgjOBcYCnwO4+3pgt6/UunsFcBXwIrAMeNLdl5jZHWY2NlzsRWCzmS0F5gI3ufvmPf8aiVdV5RQUblG1kIi0OU15fHSnu7uZOYCZNbnxfXefDcyuM+32mGEHrg8/KW3lxmI+K63Q+wMi0uY05YrgSTP7PdDFzL4DvAw8mNiwUk9+eH9AbxSLSFvT6BWBBY3tzwQGAZ8BA4Hb3f1frRBbSskPO6Lpc0D7ZIciItKiGk0EYZXQbHcfCkSu8I9VUFjEiP7qiEZE2p6mVA0tMLMRCY8khVV3RJOnF8lEpA1qys3iLwIXm9l/CZ4cMoKLhWEJjSyFVHdEoyeGRKQtakoiOC3hUaS4gsIidUQjIm3WbquG3P2/QBfg7PDTJZwWGfmFW9QRjYi0WU15s/ha4HHgwPDzmJldnejAUkV1RzSqFhKRtqopVUNXAF90988BzOxO4C3g/kQGlioWfFQUdkSjG8Ui0jY1pa7DgMqY8cpwWiTkF24hI80Y3keJQETapqZcEfwR+I+Z/TUcPwf4Q+JCSi0FhUXqiEZE2rSm3Cz+JXAZsCX8XObuUxMdWCooq6hk4dqt6ohGRNq03V4RmNmxwBJ3XxCO72dmX3T3/yQ8uiRbtHYbOyuq1NCciLRpTblHMB0ojhkvDqe1efmFRQDkqSMaEWnDmnSzOGwuGgB3r6Jp9xb2eQWFWzhEHdGISBvXlETwgZldY2btws+1pHAvYi2lqsop+G+Rmp0WkTavKYngSuBLBP0NryNoe2hCIoNKBSs3FrOtpFz3B0SkzdttFY+7byTobzhS8msamtP9ARFp2xq8IjCz75jZYeGwmdnDZrbNzN4zs6NbL8TkKCjcQvecLPoe0CHZoYiIJFRjVUPXAoXh8HjgKOAQgv6F70tsWMmXX1jEiP77qyMaEWnzGksEFe5eHg6fBTzi7pvd/WWgyR3Y74vWhx3RqKE5EYmCxhJBlZn1MLNs4CsEndZXa9Md9+arIxoRiZDGbhbfDhQA6cAsd18CYGYn0sYfHy0oLKJjZro6ohGRSGgwEbj782bWD8hx96KYWQXAhQmPLInyC7dwdL/91RGNiERCoyWdu1fUSQK4++fuXtzQOvu6bSVBRzR5/VQtJCLRoFPeOqo7ohkxQO8PiEg0KBHUkf9h0BFNbp8uyQ5FRKRVNCsRmNmglg4kVRQUFjG4V2c6ZEaiXT0RkWZfEbzUolGkiJqOaNTstIhESIOnvWb264ZmAW2y3mTxOnVEIyLR01j9x2XADUBZnHnjExNOclV3RKOG5kQkShpLBPnAYnd/s+4MM5uSsIiSKP9DdUQjItHTWCI4HyiNN8PdByQmnOSp7ohmzOCDkx2KiEirauxmcSd339FqkSTZqk3VHdGoWkhEoqWxRPC36gEze6YVYkkqNTQnIlHVWCKIbYj/kOZs3MzGmNlyM1tlZrc0stzXzMzNLK85+2kJ+R8GHdH066qOaEQkWhpLBN7AcJOYWTowDTgdOBIYb2ZHxlkuh6ATnP/s6T5akjqiEZGoaiwRHGVmn5nZdmBYOPyZmW03s8+asO2RwCp3/8DddwJPAOPiLPdj4E4auDHdGqo7olFDcyISRQ0mAndPd/f93D3H3TPC4erx/Zqw7V7AmpjxteG0GmHfx33c/R+NbcjMJphZgZkVbNq0qQm73jMF/61+f0CJQESiJ2mNzplZGvBLgpfWGuXuD7h7nrvnde/evcVjKSjcQsfMdI7ooY5oRCR6EpkI1gF9YsZ7h9Oq5QBDgHlmVggcC8xKxg3jtz9URzQiEl2JLPnygcPMbICZZQLfAGZVz3T3be7ezd37u3t/4N/AWHcvSGBM9agjGhGJuoQlAnevAK4CXgSWAU+6+xIzu8PMxiZqv3uqpiMavUgmIhGV0Eb33X02MLvOtNsbWHZ0ImNpSEFh2BFN3zbZoKqIyG5FvlI8Xx3RiEjERToRlFVUsnCNOqIRkWiLdCJQRzQiIhFPBNUd0ajFURGJskgngoLCLRzSrSPd1BGNiERYZBNBdUc0alZCRKIusolg1aZitu5QRzQiIpFNBOqIRkQkENlEUFBYRLdO6ohGRCSyiSC/cIs6ohERIaKJYMO2EtYWlahaSESEiCaC6vcHlAhERCKaCAoKt9BBHdGIiAARTQT5hUUc3Vcd0YiIQAQTwWel5bz/8Wd6f0BEJBS5RLDgv0FHNCN1f0BEBIhgIsgv3EK6OqIREakRwURQxJCe+6kjGhGRUKQSQVlFJe+u2ar+B0REYkQqESxe9xllFVXqqF5EJEakEkFB2NCcrghERHaJVCLIV0c0IiL1RCYRVHdEo/cHRERqi0wiWF3TEY2qhUREYkUmEaihORGR+CKTCPoc0J4L8nrTXx3RiIjUEpm3qkYd1p1Rh3VPdhgiIiknMlcEIiISnxKBiEjEKRGIiEScEoGISMQpEYiIRJwSgYhIxCkRiIhEXEITgZmNMbPlZrbKzG6JM/96M1tqZu+Z2Rwz65fIeEREpL6EJQIzSwemAacDRwLjzezIOou9A+S5+zDgaeCuRMUjIiLxJfKKYCSwyt0/cPedwBPAuNgF3H2uu+8IR/8N9E5gPCIiEkciE0EvYE3M+NpwWkOuAF6IN8PMJphZgZkVbNq0qQVDFBGRlLhZbGb/A+QBd8eb7+4PuHueu+d17672gkREWlIiG51bB/SJGe8dTqvFzL4K3Aac6O5lCYxHRETiSOQVQT5wmJkNMLNM4BvArNgFzGw48HtgrLtvTGAsIiLSgIQlAnevAK4CXgSWAU+6+xIzu8PMxoaL3Q10Ap4ys4VmNquBzYmISIIktD8Cd58NzK4z7faY4a8mcv8iIrJ7kemYRiSKysvLWbt2LaWlpckORVpJdnY2vXv3pl27dk1eR4lApA1bu3YtOTk59O/fHzNLdjiSYO7O5s2bWbt2LQMGDGjyeinx+KiIJEZpaSldu3ZVEogIM6Nr1657fAWoRCDSxikJREtzft9KBCIiEadEICIJ89e//pXc3Nxan7S0NF54IW5rMi1u3rx5nHXWWXu0ztSpU9mxY0fcea+99hqDBw8mNzeXkpKSlgixSTp16pTQ7SsRiEjCnHvuuSxcuLDmM2nSJEaNGsVpp53WpPXdnaqqqgRHWVtjieDxxx/n1ltvZeHChbRv375mekVFRWuFlxBKBCJRMnp0/c9vfxvM27Ej/vwZM4L5n35af94eWLFiBXfccQePPvooaWlB0XP33XczYsQIhg0bxg9/+EMACgsLGThwIN/61rcYMmQIa9as4aabbmLIkCEMHTqUmTNnxt3+K6+8UnPVMXz4cLZv3w5AcXEx559/PoMGDeLiiy/G3QGYM2cOw4cPZ+jQoVx++eWUlZXx61//mvXr13PSSSdx0kkn1dr+Qw89xJNPPsnkyZO5+OKLmTdvHqNGjWLs2LEceeSRlJaWctlllzF06FCGDx/O3LlzAZgxYwbnnHMOp5xyCv379+c3v/kNv/zlLxk+fDjHHnssW7ZsqfddPvzwQ4477jiGDh3KD37wg1rz4h2zvaVEICIJV15ezkUXXcS9995L3759AXjppZdYuXIlb7/9NgsXLmT+/Pm8+uqrAKxcuZJJkyaxZMkSCgoKWLhwIe+++y4vv/wyN910Exs2bKi3j3vuuYdp06axcOFCXnvttZoz9nfeeYepU6eydOlSPvjgA9544w1KS0u59NJLmTlzJosWLaKiooLp06dzzTXX0LNnT+bOnVtTkFf79re/zdixY7n77rt5/PHHAViwYAH33XcfK1asYNq0aZgZixYt4i9/+QuXXHJJzdM7ixcv5tlnnyU/P5/bbruNDh068M4773DcccfxyCOP1Psu1157LRMnTmTRokX06NGjZnpjx2xv6D0CkSiZN6/heR06ND6/W7fG5zdi8uTJDB48mAsvvLBm2ksvvcRLL73E8OHDgeDMfeXKlfTt25d+/fpx7LHHAvD6668zfvx40tPTOeiggzjxxBPJz89n7NixtfZx/PHHc/3113PxxRdz3nnn0bt30L3JyJEja4Zzc3MpLCwkJyeHAQMGcPjhhwNwySWXMG3aNK677ro9+l4jR46seV7/9ddf5+qrrwZg0KBB9OvXjxUrVgBw0kknkZOTQ05ODp07d+bss88GYOjQobz33nv1tvvGG2/wzDPPAPDNb36Tm2++udFjdsIJJ+xR3HUpEYhIQs2bN49nnnmGBQsW1Jru7tx6661897vfrTW9sLCQjh077na706ZN48EHHwRg9uzZ3HLLLZx55pnMnj2b448/nhdffBGArKysmnXS09NbtD6/KXHWjSEtLa1mPC0trcF44j0G2tAx21uqGhKRhCkqKuKyyy7jkUceIScnp9a80047jYcffpji4mIA1q1bx8aN9RshHjVqFDNnzqSyspJNmzbx6quvMnLkSL73ve/V3ITu2bMnq1evZujQodx8882MGDGC999/v8G4Bg4cSGFhIatWrQLg0Ucf5cQTTwQgJyen5v7Cnhg1alRNldGKFSv46KOPGDhw4B5vB4KrmyeeeAKgZpvQ9GO2p3RFICIJ87vf/Y6NGzcyceLEWtNvvfVWLrzwQpYtW8Zxxx0HBI9IPvbYY6Snp9da9txzz+Wtt97iqKOOwsy46667OPjgg+vta+rUqcydO5e0tDQGDx7M6aefzltvvRU3ruzsbP74xz/y9a9/nYqKCkaMGMGVV14JwIQJExgzZkzNvYKmmjRpEhMnTmTo0KFkZGQwY8aMWlcCe+K+++7joosu4s4772TcuF09/J566qlxj9mBBx7YrP1Us+o76PuKvLw8LygoSHYYIvuEZcuWccQRRyQ7DGll8X7vZjbf3fPiLa+qIRGRiFMiEBGJOCUCEZGIUyIQEYk4JQIRkYhTIhARiTglAhFJqPT09FrNUBcWFvKlL32pRba9detWflvdaF6CjR49mkQ8uj5+/HiGDRvGr371qxbfdlPphTIRSaj27duzcOHCWtPefPPNFtl2dSKYNGlSi2yvtX388cfk5+fXvOEcq6KigoyM1imilQhEIuJHzy1h6frPWnSbR/bcjx+ePXiP1+vUqRPFxcXMmzePKVOm0K1bNxYvXswxxxzDY489hpkxf/58rr/+eoqLi+nWrRszZsyo1RInwC233MLq1avJzc3llFNO4cwzz+See+7h+eefB+Cqq64iLy+PSy+9lP79+3PJJZfw3HPPUV5ezlNPPcWgQYP4/PPPufrqq1m8eDHl5eVMmTKFcePGUVJSwmWXXca7777LoEGDGuyIZs6cOdx44401byhPnz6drKysBvcX69RTT2XdunXk5uZy//33M3nyZHJzc2sa2rvhhhv2+Ng2h6qGRCShSkpKaqqFzj333Hrz4zUTXV5eztVXX83TTz/N/Pnzufzyy7ntttvqrfuLX/yCQw89lIULF3L33XfvNpZu3bqxYMECJk6cyD333APAT3/6U04++WTefvtt5s6dy0033cTnn3/O9OnT6dChA8uWLeNHP/oR8+fPr7e9hpqzbmx/sWbNmlUT/6hRowDYuXMnBQUFrZYEQFcEIpHRnDP3lhCvaihWvGaiu3TpwuLFiznllFMAqKysrHc10BznnXceAMcccwzPPvssEDTtPGvWrJqCurS0lI8++ohXX32Va665BoBhw4YxbNiwettbvnx5o81Zx9vf7sQ21d1alAhEJKniNRPt7gwePLheo3Fr1qypacv/yiuvZMyYMbXmZ2Rk1OrasrpjmLr7im2O2t155plnmt1SaGPi7W93mtq0dUtS1ZCIpJyBAweyadOmmkRQXl7OkiVL6NOnT03T01deeWW9JqP79evH0qVLKSsrY+vWrcyZM2e3+zrttNO4//77a7qwfOeddwA44YQT+POf/wwEPYzF60Cmseas9yVKBCKScjIzM3n66ae5+eabOeqoo8jNzY37pFHXrl05/vjjGTJkCDfddBN9+vThggsuYMiQIVxwwQU1PXk1ZvLkyZSXlzNs2DAGDx7M5MmTAZg4cSLFxcUcccQR3H777RxzzDH11o1tznro0KGkpaXVNGe9L1Ez1CJtmJqhjiY1Qy0iIntEiUBEJOKUCETauH2t+lf2TnN+30oEIm1YdnY2mzdvVjKICHdn8+bNZGdn79F6eo9ApA3r3bs3a9euZdOmTckORVpJdnZ2zQt6TaVEINKGtWvXjgEDBiQ7DElxCa0aMrMxZrbczFaZ2S1x5meZ2cxw/n/MrH8i4xERkfoSlgjMLB2YBpwOHAmMN7Mj6yx2BVDk7l8AfgXcmah4REQkvkReEYwEVrn7B+6+E3gCGFdnmXHAn8Lhp4GvmJklMCYRESS+zaUAAArCSURBVKkjkfcIegFrYsbXAl9saBl3rzCzbUBX4NPYhcxsAjAhHC02s+UJibh5ulEn3hST6vFB6seY6vFB6seY6vFB24+xX0Mz9ombxe7+APBAsuOIx8wKGnptOxWkenyQ+jGmenyQ+jGmenwQ7RgTWTW0DugTM947nBZ3GTPLADoDmxMYk4iI1JHIRJAPHGZmA8wsE/gGMKvOMrOAS8Lh84H/c735IiLSqhJWNRTW+V8FvAikAw+7+xIzuwMocPdZwB+AR81sFbCFIFnsa1KyyipGqscHqR9jqscHqR9jqscHEY5xn2uGWkREWpbaGhIRiTglAhGRiFMi2A0zKzSzRWa20MwKwmkHmNm/zGxl+HP/cLqZ2a/DJjPeM7OjExTTw2a20cwWx0zb45jM7JJw+ZVmdkm8fbVgfFPMbF14HBea2Rkx824N41tuZqfFTG+0iZK9jLGPmc01s6VmtsTMrg2np8RxbCS+lDmOZpZtZm+b2bthjD8Kpw8Im4xZFTYhkxlOb7BJmYZiT1B8M8zsw5hjmBtOb/X/lZjtp5vZO2b2fDjeusfQ3fVp5AMUAt3qTLsLuCUcvgW4Mxw+A3gBMOBY4D8JiukE4GhgcXNjAg4APgh/7h8O75/A+KYAN8ZZ9kjgXSALGACsJni4ID0cPgTIDJc5sgWPYQ/g6HA4B1gRxpISx7GR+FLmOIbHolM43A74T3hsngS+EU7/HTAxHJ4E/C4c/gYws7HYExjfDOD8OMu3+v9KzL6vB/4MPB+Ot+ox1BVB88Q2jfEn4JyY6Y944N9AFzPr0dI7d/dXCZ6y2puYTgP+5e5b3L0I+BcwJoHxNWQc8IS7l7n7h8AqguZJmtJEyd7EuMHdF4TD24FlBG+6p8RxbCS+hrT6cQyPRXE42i78OHAyQZMxUP8YxmtSpqHYExVfQ1r9fwXAzHoDZwIPheNGKx9DJYLdc+AlM5tvQVMXAAe5+4Zw+GPgoHA4XrMajf3ztqQ9jSkZsV4VXnI/XF3lkgrxhZfXwwnOGFPuONaJD1LoOIZVGguBjQQF5Gpgq7tXxNlfrSZlgOomZRIWY9343L36GP40PIa/MrOsuvHViSPRv+OpwP8DqsLxrrTyMVQi2L0vu/vRBK2ofs/MToid6cF1WUo9g5uKMQHTgUOBXGADcG9ywwmYWSfgGeA6d/8sdl4qHMc48aXUcXT3SnfPJWg5YCQwKJnx1FU3PjMbAtxKEOcIguqem5MVn5mdBWx09/nJigGUCHbL3deFPzcCfyX4Y/+kuson/LkxXLwpzWokyp7G1Kqxuvsn4T9lFfAguy5bkxafmbUjKGQfd/dnw8kpcxzjxZeKxzGMayswFziOoEql+mXV2P011KRMwmOMiW9MWO3m7l4G/JHkHsPjgbFmVkhQbXcycB+tfQxb4kZHW/0AHYGcmOE3CeoG76b2DcW7wuEzqX2z6e0Extaf2jdj9ygmgjOhDwlufu0fDh+QwPh6xAz/L0F9JsBgat/k+oDgBmdGODyAXTc5B7dgfAY8AkytMz0ljmMj8aXMcQS6A13C4fbAa8BZwFPUvtE5KRz+HrVvdD7ZWOwJjK9HzDGeCvwimf8rMfGOZtfN4lY9hi36Rdrah+BJi3fDzxLgtnB6V2AOsBJ4ufqPIvwDmkZQT7oIyEtQXH8hqBYoJ6gLvKI5MQGXE9xUWgVcluD4Hg33/x5BG1OxBdptYXzLgdNjpp9B8LTM6upj34Ixfpmg2uc9YGH4OSNVjmMj8aXMcQSGAe+EsSwGbo/5v3k7PB5PAVnh9OxwfFU4/5DdxZ6g+P4vPIaLgcfY9WRRq/+v1Il3NLsSQaseQzUxISIScbpHICIScUoEIiIRp0QgIhJxSgQiIhGnRCAiEnFKBNIsZuZmdm/M+I1mNqWFtj3DzM5viW3tZj9fN7NlZjY3ZtrQmFYpt8S0UvlyE7c5dnctfJpZTzN7urFlmsrMDjKz58MWNpea2eyW2G4j++tvMa3KStuQsK4qpc0rA84zs5+7+6fJDqaamWX4rjZaducK4Dvu/nr1BHdfRNB8A2Y2g+C57lqFdmP78KAL1rp9c9ddZj1BH90t4Q6CNnTuC2Mb1kLblQjRFYE0VwVB/6n/W3dG3TN6MysOf442s1fM7O9m9oGZ/cLMLg7bjF9kZofGbOarZlZgZivC9liqGxC728zywwbDvhuz3dfMbBawNE4848PtLzazO8NptxO8tPUHM7t7d1/WzOaZ2VQL+qS41szODtuDf8fMXjazg8LlLjWz38Qch1+b2Zvh9z0/nF5zVh0u/6yZ/dOCtu7vitnnFeH3f9vMHqzebh09CF7aA8Dd3wvX7WRmc8xsQfjdx8Xs+/0wthVm9riZfdXM3gj3PzJcboqZPWpmb4XTvxPnmDT0++hhZq+GV1KLzWzU7o6vJJeuCGRvTAPeiy28muAo4AiCZqo/AB5y95EWdLxyNXBduFx/gjZgDgXmmtkXgG8B29x9hAUtRr5hZi+Fyx8NDPGgCd4aZtYTuBM4BigiaEn2HHe/w8xOJmjbv6CJsWe6e1643f2BY93dzezbBK1H3hBnnR4ECWcQwZVCvCqhXILWRcuA5WZ2P1AJTA6/13aCt2HfjbPuNGCmmV1F8Cb0H8MrjlLgXHf/zMy6Af8OEyXAF4CvE7wtmw9cFMY4Fvg+u5o8HkbQ1EJH4B0z+0edfV9B/N/HecCL7v5TM0sHOsSJW1KIEoE0W1jIPAJcA5Q0cbV8D5t5NrPVQHVBvgg4KWa5Jz1oWG2lmX1AUJCeCgyLudroDBwG7CRoF6ZWEgiNAOa5+6Zwn48TdJzztybGG2tmzHBvggK4B0EbPvH2DfC38Hssrb5qiGOOu28L41sK9AO6Aa+4+5Zw+lPA4XVXdPcXzewQgjawTicosIcAW4GfWdBabhVBk8TV+/8wrALDzJaE+3czW0SQgKv93d1LgBIL7qOMJGjqolpDv4984GELGs37m7vHriMpSIlA9tZUYAFBK47VKgirHc0sjaCgrFYWM1wVM15F7b/Hum2fOEFbMFe7+4uxM8xsNPB588LfI7H7uB/4pbvPCvc/pYF1Yr+vNWGZSvbw/zJMFn8G/mxBV4cnEPRq1h04xt3LLWjdMjvO/vb0dxAr7u8DIExAZwIzzOyX7v7InnwnaV26RyB7JSyEniSoJqhWSFAVA0F1Q7tmbPrrZpYW3jc4hKAhrReBieGZJmZ2uJl13M123gZONLNuYTXFeOCVZsRTV2d2NfN7SQtsr658grj3t6C54a/FW8jMTjazDuFwDkFV2kdhfBvDJHASwVXGnhpnQb+/XQkaRMuvMz/u78PM+gGfuPuDBL1uJaTvbmk5uiKQlnAvcFXM+IPA383sXeCfNO9s/SOCQnw/4Ep3LzWzhwiqLhaYmQGb2FWfHZe7b7Dgcc65BGew/3D3vzcjnrqmAE+ZWRFB/f2AFthmDXdfZ2Y/IzgGW4D3CXqjqusY4DdmVn0V9pC755vZh8BzYXVPQbj+nnqP4Lh1A37s7ustprN0gkK+P/V/H6OBm8ysHCgmuLcjKUytj4qkKDPr5O7F4RXBX4GH3f2vrbTvKUCxu9/TGvuT5FLVkEjqmmJBf7uLCW5GN+cGt8hu6YpARCTidEUgIhJxSgQiIhGnRCAiEnFKBCIiEadEICIScf8f/H9WaZVPEq0AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax.axhline(f1_scores[\"de\"][\"fr\"], ls=\"--\", color=\"r\")\n",
    "metrics_df.set_index(\"num_samples\").plot(ax=ax)\n",
    "plt.legend([\"Zero-shot from de\", \"Fine-tuned on fr\"], loc=\"lower right\")\n",
    "plt.ylim((0, 1))\n",
    "plt.xlabel(\"Number of Training Samples\")\n",
    "plt.ylabel(\"F1 Score\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "id": "dwPMAlOg3pUw",
    "outputId": "7fbbebbd-9b12-415f-9d96-6cd7e155f9e9"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "\" DatasetDict({\\n    train: Dataset({\\n        features: ['input_ids', 'attention_mask', 'labels'],\\n        num_rows: 12580\\n    })\\n    validation: Dataset({\\n        features: ['input_ids', 'attention_mask', 'labels'],\\n        num_rows: 6290\\n    })\\n    test: Dataset({\\n        features: ['input_ids', 'attention_mask', 'labels'],\\n        num_rows: 6290\\n    })\\n}) \""
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "panx_de_encoded\n",
    "\n",
    "''' DatasetDict({\n",
    "    train: Dataset({\n",
    "        features: ['input_ids', 'attention_mask', 'labels'],\n",
    "        num_rows: 12580\n",
    "    })\n",
    "    validation: Dataset({\n",
    "        features: ['input_ids', 'attention_mask', 'labels'],\n",
    "        num_rows: 6290\n",
    "    })\n",
    "    test: Dataset({\n",
    "        features: ['input_ids', 'attention_mask', 'labels'],\n",
    "        num_rows: 6290\n",
    "    })\n",
    "}) '''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wDVwlN8U3pUx",
    "outputId": "93ee7cff-cdfc-4756-9154-76e0b14b1178"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['train', 'validation', 'test'])"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "panx_de_encoded.keys()\n",
    "# dict_keys(['train', 'validation', 'test'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "id": "EK4NIBuu3pUx"
   },
   "outputs": [],
   "source": [
    "from datasets import concatenate_datasets\n",
    "\n",
    "def concatenate_splits(corpora):\n",
    "    \"\"\"\n",
    "    Function to concatenate multiple corpora together, and shuffle the result.\n",
    "\n",
    "    Parameters:\n",
    "    corpora (list): A list of corpora, where each corpus is an instance of DatasetDict from the HuggingFace `datasets` library.\n",
    "                     Each DatasetDict should contain the same train/validation/test splits.\n",
    "\n",
    "    Returns:\n",
    "    multi_corpus (DatasetDict): A DatasetDict containing the concatenated and shuffled datasets.\n",
    "    \"\"\"\n",
    "\n",
    "    # Initialize an empty DatasetDict to hold the concatenated datasets.\n",
    "    multi_corpus = DatasetDict()\n",
    "\n",
    "    # For each split (train, validation, test) in the first corpus in the list...\n",
    "    for train_val_test_split in corpora[0].keys():\n",
    "\n",
    "        # Concatenate the corresponding split from all corpora in the list.\n",
    "        # The `concatenate_datasets` function from HuggingFace `datasets` is used here.\n",
    "        # The concatenated dataset is then shuffled with a fixed seed for reproducibility.\n",
    "        multi_corpus[train_val_test_split] = concatenate_datasets(\n",
    "            [corpus[train_val_test_split] for corpus in corpora]).shuffle(seed=42)\n",
    "\n",
    "    # Return the concatenated and shuffled DatasetDict.\n",
    "    return multi_corpus\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "id": "kjsL3shE3pUx"
   },
   "outputs": [],
   "source": [
    "panx_de_fr_concatenated_encoded = concatenate_splits([panx_de_encoded, panx_fr_encoded])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "nXAYEUn13pUx",
    "outputId": "72b20eca-e29e-40e9-9975-fc234c320455"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:310: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  FutureWarning,\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='2145' max='2145' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [2145/2145 11:44, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.285400</td>\n",
       "      <td>0.183071</td>\n",
       "      <td>0.823534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.149700</td>\n",
       "      <td>0.159730</td>\n",
       "      <td>0.848593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.096100</td>\n",
       "      <td>0.160419</td>\n",
       "      <td>0.860103</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 2.1377685e+00 -7.3056984e-01 -1.2598184e+00 ... -4.7928718e-01\n",
      "   -5.4880399e-01 -1.4730066e-01]\n",
      "  [ 8.2309427e+00 -2.0092292e+00 -2.3204882e+00 ... -1.3796846e+00\n",
      "   -1.2631409e+00 -1.4006596e+00]\n",
      "  [ 8.2432051e+00 -2.4137337e+00 -2.0702550e+00 ... -8.5598320e-01\n",
      "   -1.8062910e+00 -9.0289891e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 2.5173934e+00 -2.0759494e+00 -2.6205986e+00 ... -1.7687498e+00\n",
      "    2.2808824e+00 -1.9617949e-01]\n",
      "  [ 6.5400515e+00 -1.7168512e+00 -2.6269498e+00 ... -1.7725747e+00\n",
      "    2.3659371e-01 -1.9938818e+00]\n",
      "  [ 7.3535876e+00 -2.3388042e+00 -2.3144970e+00 ... -8.2872742e-01\n",
      "   -1.1674420e+00 -1.3158110e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.5530260e+00 -5.7519746e-01 -1.0172466e+00 ... -1.0932462e+00\n",
      "    1.4131622e-01 -1.3806552e+00]\n",
      "  [ 7.6467519e+00 -2.6194911e+00 -1.2325330e+00 ... -9.7952455e-01\n",
      "   -1.9023544e+00 -1.1428338e+00]\n",
      "  [ 7.5975156e+00 -2.4881673e+00 -1.3742369e+00 ... -9.5412272e-01\n",
      "   -1.8357586e+00 -1.2339892e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 2.1799207e+00 -6.9757128e-01 -1.1951884e+00 ... -3.6695984e-01\n",
      "   -7.2073382e-01 -4.0850721e-02]\n",
      "  [ 8.2629404e+00 -2.2489891e+00 -1.7607373e+00 ... -8.9741635e-01\n",
      "   -1.7701259e+00 -9.8101294e-01]\n",
      "  [ 8.3289671e+00 -2.4365726e+00 -1.6661239e+00 ... -6.4766830e-01\n",
      "   -2.0663760e+00 -7.5866181e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.0506883e+00 -1.4119385e+00 -1.8853383e+00 ... -9.4712859e-01\n",
      "   -3.2865137e-02  2.5855340e-02]\n",
      "  [ 8.3177919e+00 -2.1974587e+00 -2.1245384e+00 ... -1.3311447e+00\n",
      "   -1.1821599e+00 -1.1105174e+00]\n",
      "  [ 8.4270334e+00 -2.3579683e+00 -2.1105740e+00 ... -1.1522897e+00\n",
      "   -1.3844143e+00 -7.9611444e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.0491526e+00 -1.0761408e+00 -1.7117075e+00 ... -8.4414482e-01\n",
      "   -3.3888724e-01 -3.4915039e-01]\n",
      "  [ 8.2123899e+00 -1.9458013e+00 -1.8244374e+00 ... -1.0844355e+00\n",
      "   -1.5661178e+00 -1.1560764e+00]\n",
      "  [ 8.4199677e+00 -2.1535997e+00 -2.0425334e+00 ... -8.9185691e-01\n",
      "   -1.7131764e+00 -8.8932598e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n",
      "eval_pred.predictions  [[[ 3.8141294e+00 -8.8258141e-01 -1.7512832e+00 ... -6.8298972e-01\n",
      "   -6.2518942e-01 -1.3195244e+00]\n",
      "  [ 8.5361834e+00 -1.7231473e+00 -2.0101645e+00 ... -1.3532228e+00\n",
      "   -1.3265795e+00 -2.4309170e+00]\n",
      "  [ 8.5957060e+00 -2.0419760e+00 -1.8644830e+00 ... -1.0190141e+00\n",
      "   -1.7119242e+00 -2.0498149e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.2659500e+00 -1.9244945e+00 -2.3994091e+00 ... -2.1022117e+00\n",
      "    1.5359168e+00  2.1556634e-01]\n",
      "  [ 6.9675226e+00 -1.4004700e+00 -2.8777432e+00 ... -2.2593260e+00\n",
      "    5.4824156e-01 -2.8177679e+00]\n",
      "  [ 7.9088240e+00 -1.8133434e+00 -2.4349167e+00 ... -1.4043599e+00\n",
      "   -9.4678581e-01 -2.3561237e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 2.3802493e+00  7.3822844e-01  7.0386451e-01 ... -1.2124634e+00\n",
      "   -6.1479419e-01 -3.1182997e+00]\n",
      "  [ 7.3987193e+00 -1.6601491e+00 -1.1756313e+00 ... -9.7912014e-01\n",
      "   -2.0287263e+00 -2.3613698e+00]\n",
      "  [ 7.6202636e+00 -1.8210899e+00 -9.9466652e-01 ... -7.3420703e-01\n",
      "   -2.2333395e+00 -2.3613884e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 3.3399611e+00 -6.1506134e-01 -1.2045645e+00 ... -4.2509949e-01\n",
      "   -1.0616820e+00 -7.5341529e-01]\n",
      "  [ 8.6045799e+00 -1.7275505e+00 -1.7104536e+00 ... -1.0257150e+00\n",
      "   -1.5928185e+00 -2.2167630e+00]\n",
      "  [ 8.6842785e+00 -2.0650368e+00 -1.3070327e+00 ... -6.1440766e-01\n",
      "   -2.0900352e+00 -1.8761926e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.4692309e+00 -1.3911803e+00 -1.8764015e+00 ... -1.1884866e+00\n",
      "    1.8158261e-01 -1.1618225e-01]\n",
      "  [ 8.6243744e+00 -1.9081577e+00 -2.1104932e+00 ... -1.6438448e+00\n",
      "   -8.3850867e-01 -2.2655756e+00]\n",
      "  [ 8.7647028e+00 -2.1666083e+00 -1.8849621e+00 ... -1.2982290e+00\n",
      "   -1.2048119e+00 -1.8202691e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 5.4516768e+00 -1.3416513e+00 -2.2806499e+00 ... -1.4900644e+00\n",
      "   -2.0641202e-02 -1.7533798e+00]\n",
      "  [ 8.7733831e+00 -1.7481251e+00 -1.7319089e+00 ... -1.1665779e+00\n",
      "   -1.5174078e+00 -2.3631587e+00]\n",
      "  [ 8.8701601e+00 -1.9347765e+00 -1.7230860e+00 ... -9.0910739e-01\n",
      "   -1.7465858e+00 -1.9887663e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n",
      "eval_pred.predictions  [[[ 2.6261714e+00 -6.8262577e-01 -1.4115199e+00 ... -6.5837294e-01\n",
      "   -2.7426264e-01 -7.9443365e-01]\n",
      "  [ 8.8138971e+00 -1.7837485e+00 -2.0911894e+00 ... -1.7040812e+00\n",
      "   -1.1011778e+00 -2.4456985e+00]\n",
      "  [ 8.9195309e+00 -2.1015673e+00 -1.9601572e+00 ... -1.3989074e+00\n",
      "   -1.4702951e+00 -2.0572569e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 2.1757646e+00 -1.9684669e+00 -2.5128171e+00 ... -2.2457154e+00\n",
      "    2.3994102e+00  1.1955864e+00]\n",
      "  [ 8.2053404e+00 -1.9230345e+00 -2.7682180e+00 ... -2.3170648e+00\n",
      "    7.5814188e-02 -2.5333745e+00]\n",
      "  [ 8.7062578e+00 -1.9965417e+00 -2.3148615e+00 ... -1.6887053e+00\n",
      "   -1.1317457e+00 -2.2813132e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 7.9558080e-01  1.6799964e+00  1.6772017e+00 ... -1.0492710e+00\n",
      "   -8.5063213e-01 -2.6745317e+00]\n",
      "  [ 8.2384272e+00 -1.9823308e+00 -9.7116601e-01 ... -1.1507818e+00\n",
      "   -2.2865794e+00 -2.3295755e+00]\n",
      "  [ 8.3522778e+00 -2.0006535e+00 -8.1819195e-01 ... -1.0347525e+00\n",
      "   -2.3164446e+00 -2.3492477e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 2.6926835e+00 -5.5736303e-01 -1.0844777e+00 ... -4.6987900e-01\n",
      "   -8.4759349e-01 -4.4548461e-01]\n",
      "  [ 8.9720135e+00 -1.9362361e+00 -1.5996745e+00 ... -1.2139285e+00\n",
      "   -1.5048661e+00 -2.0313084e+00]\n",
      "  [ 8.9757557e+00 -2.2520504e+00 -1.1793545e+00 ... -7.7435285e-01\n",
      "   -2.0281963e+00 -1.6411601e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 2.7182879e+00 -1.4624873e+00 -1.9550318e+00 ... -1.2431184e+00\n",
      "    6.9001311e-01  4.6298268e-01]\n",
      "  [ 9.0166035e+00 -2.0255146e+00 -2.1596813e+00 ... -1.9708681e+00\n",
      "   -6.0418606e-01 -2.1235571e+00]\n",
      "  [ 9.1726074e+00 -2.2267773e+00 -2.0139544e+00 ... -1.6529291e+00\n",
      "   -8.8657248e-01 -1.7205527e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.3287616e+00 -1.4437662e+00 -2.3799043e+00 ... -1.4612375e+00\n",
      "    6.6096765e-01 -1.0239114e+00]\n",
      "  [ 9.0407219e+00 -1.8261669e+00 -1.6733067e+00 ... -1.3520608e+00\n",
      "   -1.4019371e+00 -2.2554181e+00]\n",
      "  [ 9.2445450e+00 -2.0018370e+00 -1.6974860e+00 ... -1.1315842e+00\n",
      "   -1.5813987e+00 -1.9412102e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=2145, training_loss=0.17707352360367498, metrics={'train_runtime': 704.6539, 'train_samples_per_second': 73.057, 'train_steps_per_second': 3.044, 'total_flos': 1139987211037488.0, 'train_loss': 0.17707352360367498, 'epoch': 3.0})"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "training_args.logging_steps = len(panx_de_fr_concatenated_encoded[\"train\"]) // batch_size\n",
    "# training_args.push_to_hub = True\n",
    "training_args.push_to_hub = False  # PAUL - Changing to False\n",
    "training_args.output_dir = \"xlm-roberta-base-finetuned-panx-de-fr\"\n",
    "\n",
    "trainer = Trainer(model_init=model_init, args=training_args,\n",
    "    data_collator=data_collator, compute_metrics=compute_metrics,\n",
    "    tokenizer=xlmr_tokenizer, train_dataset=panx_de_fr_concatenated_encoded[\"train\"],\n",
    "    eval_dataset=panx_de_fr_concatenated_encoded[\"validation\"])\n",
    "\n",
    "trainer.train()\n",
    "# trainer.push_to_hub(commit_message=\"Training completed!\")  # PAUL - COMMENTING OUT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "referenced_widgets": [
      "edd5de97408a4760a2e770e49c189246",
      "a87b0705985046cf9abb1e906d7d196b",
      "7a5419c120804ae9bc3618f6c5dea22d",
      "ef43aed878ca459e96b039d44e4f1cbd",
      "227468d815d049bfa9d9b11dd1d502de",
      "0dfd91a2f6a8481b87bddd642feb3e06",
      "0a95843a9a284cc2990947038bd17bf0",
      "dd6e54ead891406ebbbd20a308fb6e56",
      "75e551bbe19941079a3cb8e10bd2bd92",
      "af02475f27bb4c24913a7050d6fe61d2",
      "3ce86a1149b14240a998541d525c41f2"
     ]
    },
    "id": "QjrVnxjt3pUx",
    "outputId": "39ec614b-1400-420a-8d52-028525566ce1"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "edd5de97408a4760a2e770e49c189246",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/13 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.de/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-7d24f9a603566431.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.de/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-c1c2d298fc80b661.arrow\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[   1.3811793    -1.4601814    -1.4864641  ...   -0.7152105\n",
      "      0.45609555    1.3456137 ]\n",
      "  [   8.85643      -1.907472     -1.8204554  ...   -1.5364349\n",
      "     -1.2552114    -1.9830589 ]\n",
      "  [   8.842626     -1.9747117    -1.6426075  ...   -1.4968807\n",
      "     -1.277305     -1.7088412 ]\n",
      "  ...\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]]\n",
      "\n",
      " [[   3.9538596     0.23638825   -0.40136725 ...   -0.14519846\n",
      "     -1.6884784    -2.9463322 ]\n",
      "  [   9.100584     -1.7412996    -1.5358851  ...   -1.38766\n",
      "     -1.6375059    -2.2748406 ]\n",
      "  [  -3.0636625     7.0976853    -1.1747248  ...   -1.3090739\n",
      "     -0.2564517    -3.469692  ]\n",
      "  ...\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]]\n",
      "\n",
      " [[   2.6307998     1.9069653    -0.28186437 ...   -1.514025\n",
      "     -0.39929768   -3.2442346 ]\n",
      "  [   7.381377     -1.1831591    -1.3912189  ...   -1.2442881\n",
      "     -1.4610523    -2.9056787 ]\n",
      "  [   7.173034     -0.7988079    -1.5271945  ...   -1.412587\n",
      "     -1.1641331    -2.6953754 ]\n",
      "  ...\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[   3.1167543    -1.5668975    -2.0734632  ...    0.94246763\n",
      "     -0.7370617    -2.3348644 ]\n",
      "  [   8.890757     -1.6393964    -1.9193811  ...   -1.5603803\n",
      "     -1.119051     -2.644158  ]\n",
      "  [   8.880597     -1.7958579    -1.9248105  ...   -1.7552629\n",
      "     -1.0711554    -2.2664711 ]\n",
      "  ...\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]]\n",
      "\n",
      " [[   4.4347568    -1.462456     -2.2043707  ...   -0.7758796\n",
      "     -0.3489396    -1.8614752 ]\n",
      "  [   8.110142     -1.7480631    -2.4890833  ...   -1.9538792\n",
      "     -0.4302285    -1.922109  ]\n",
      "  [   5.8086724    -1.5451609    -3.0963275  ...   -2.5607388\n",
      "      1.8896071    -1.05357   ]\n",
      "  ...\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]]\n",
      "\n",
      " [[   0.67612803   -1.2942224    -1.5804228  ...   -0.99343324\n",
      "      1.2773253     1.1329595 ]\n",
      "  [  -1.9852785    -1.0344847    -3.0014186  ...   -1.4540073\n",
      "      7.1114044    -0.37050182]\n",
      "  [  -1.5743572    -3.496863     -0.9133368  ...    0.20144233\n",
      "      1.0994446     6.651297  ]\n",
      "  ...\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.fr/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-92119c01382f718a.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.fr/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-df08c9bf8a078f41.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.fr/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-5a8067996ffe172e.arrow\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1-score of [de-fr] model on [de] dataset: 0.868\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[   0.13365936    2.1108885     2.0126965  ...   -0.7285474\n",
      "     -0.93652236   -3.0824475 ]\n",
      "  [   8.304991     -1.8576213    -1.0123997  ...   -1.3043203\n",
      "     -2.0060797    -2.4706943 ]\n",
      "  [   8.309917     -1.8949113    -0.9400138  ...   -1.1204985\n",
      "     -1.8162805    -2.4375947 ]\n",
      "  ...\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]]\n",
      "\n",
      " [[   1.7909784    -1.1475158    -1.8909172  ...   -1.2611046\n",
      "      1.515957     -0.2642471 ]\n",
      "  [   7.754794     -1.6979463    -1.876584   ...   -2.1544135\n",
      "     -0.64563334   -2.5751657 ]\n",
      "  [   7.795175     -2.0806787    -1.706418   ...   -1.8553467\n",
      "     -0.8363235    -2.3637307 ]\n",
      "  ...\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]]\n",
      "\n",
      " [[   2.7578423     2.0828826     2.0470154  ...   -1.6964148\n",
      "     -1.3833915    -3.41071   ]\n",
      "  [   9.051239     -1.5666403    -1.5565152  ...   -1.5360701\n",
      "     -1.6162754    -2.3813713 ]\n",
      "  [   8.574109     -0.7634358    -2.2331889  ...   -1.9018995\n",
      "     -1.0745679    -2.5823555 ]\n",
      "  ...\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[  -1.3581023    -1.7480438    -1.3089241  ...    3.3182356\n",
      "     -0.36152315   -0.4165744 ]\n",
      "  [   7.8822737    -1.6737673    -1.1619148  ...   -1.1121292\n",
      "     -2.1360333    -2.606515  ]\n",
      "  [  -1.6727253     1.3923932    -3.142526   ...   -1.3957771\n",
      "      2.6216466    -3.208957  ]\n",
      "  ...\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]]\n",
      "\n",
      " [[   0.22663555   -1.7002385    -0.61083794 ...    3.4351776\n",
      "     -1.453266     -2.147527  ]\n",
      "  [  -2.321429      1.5963488    -3.0574334  ...   -0.80172586\n",
      "      1.3341342    -3.6960711 ]\n",
      "  [  -2.5703328    -2.3089697     0.20159602 ...    6.361052\n",
      "     -2.840424     -0.679902  ]\n",
      "  ...\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]]\n",
      "\n",
      " [[   0.39959496    2.1482258     2.8604271  ...   -0.81440115\n",
      "     -1.5304545    -2.9813035 ]\n",
      "  [   8.462786     -1.9850893    -0.8930788  ...   -1.170628\n",
      "     -2.1386385    -2.3051684 ]\n",
      "  [   8.445015     -2.0192754    -0.8203366  ...   -0.991495\n",
      "     -2.0852616    -2.3099334 ]\n",
      "  ...\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]\n",
      "  [-100.         -100.         -100.         ... -100.\n",
      "   -100.         -100.        ]]]\n",
      "F1-score of [de-fr] model on [fr] dataset: 0.858\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.it/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-d1da61ce8ecf686a.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.it/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-e071a6993a6540a5.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.it/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-dc3ad1640cb4e180.arrow\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 2.6234918e+00 -1.7722435e+00 -2.3634210e+00 ...  4.0771198e-01\n",
      "   -8.9646488e-02 -9.7554380e-01]\n",
      "  [ 8.8013525e+00 -2.0324528e+00 -1.6802101e+00 ... -1.8695476e+00\n",
      "   -8.7998730e-01 -2.1281977e+00]\n",
      "  [ 8.4929924e+00 -2.1782925e+00 -1.9036101e+00 ... -1.5718662e+00\n",
      "   -7.9952896e-01 -1.5392929e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.0252202e+00 -1.8215035e+00 -2.0893064e+00 ... -1.3164822e+00\n",
      "    7.3975354e-01  4.8061293e-01]\n",
      "  [ 8.8729744e+00 -2.1427836e+00 -2.1267653e+00 ... -1.8493363e+00\n",
      "   -8.4754431e-01 -1.8851765e+00]\n",
      "  [ 8.7639751e+00 -2.3599620e+00 -1.7226259e+00 ... -1.4296449e+00\n",
      "   -1.4039841e+00 -1.4490407e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[-1.3723731e-02 -5.1769018e-01 -6.4731133e-01 ...  1.2722638e+00\n",
      "   -8.8954705e-01 -2.9469979e-01]\n",
      "  [-1.9147841e+00  2.1414380e+00 -2.6334085e+00 ... -7.3112291e-01\n",
      "    1.1812035e+00 -3.4118726e+00]\n",
      "  [-1.9257059e+00 -2.3105557e+00  1.6985484e+00 ...  5.6127896e+00\n",
      "   -2.7891312e+00  7.2661018e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[-3.5513830e-01  1.8173945e+00  2.7454600e+00 ... -8.5105532e-01\n",
      "   -9.5280927e-01 -2.3955247e+00]\n",
      "  [ 7.6354485e+00 -1.3793229e+00 -1.6728423e+00 ... -1.6625646e+00\n",
      "   -1.2535963e+00 -2.8427646e+00]\n",
      "  [ 6.7784986e+00 -1.6299803e+00 -1.6066073e+00 ... -1.7343745e+00\n",
      "   -1.3528014e+00 -2.5573976e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[-7.0066077e-01 -1.6301467e+00 -8.2368696e-01 ...  3.6995447e+00\n",
      "   -1.2461683e+00 -1.7951055e+00]\n",
      "  [-9.7009045e-01  1.3089193e-01 -3.3449848e+00 ... -5.3574942e-02\n",
      "    3.8477597e-01 -3.3839560e+00]\n",
      "  [-5.8356744e-01 -2.0251482e+00 -2.3504708e+00 ...  3.1865656e+00\n",
      "   -1.1630100e+00 -2.0668485e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[-1.6629493e+00 -1.5509969e-01  1.2863668e+00 ...  2.6102765e+00\n",
      "   -1.5613621e+00 -1.8472142e+00]\n",
      "  [-1.7235293e+00  3.0575805e+00 -2.4183240e+00 ... -1.8130175e+00\n",
      "    1.0423887e+00 -4.1913047e+00]\n",
      "  [-1.1486675e+00 -2.6385713e+00  3.0807493e+00 ...  5.2750964e+00\n",
      "   -3.4135365e+00  5.4335028e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.en/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-9c7d0d5777a90be6.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.en/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-8f6662f67966a072.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.en/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-bb9f36423604015e.arrow\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1-score of [de-fr] model on [it] dataset: 0.797\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 4.9367619e+00 -1.5472754e+00 -2.0897160e+00 ... -3.2527724e-01\n",
      "   -6.2864864e-01 -2.3695116e+00]\n",
      "  [ 9.0529938e+00 -1.9799951e+00 -1.5045127e+00 ... -1.4186020e+00\n",
      "   -1.5008993e+00 -2.2779243e+00]\n",
      "  [ 9.0453911e+00 -2.3747511e+00 -1.4446000e+00 ... -1.2523878e+00\n",
      "   -1.6009933e+00 -1.8569047e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.1097862e+00  2.0969613e+00  3.1437287e+00 ... -1.3596536e+00\n",
      "   -1.6978847e+00 -2.7673504e+00]\n",
      "  [-2.2467561e+00  8.1530390e+00 -5.0551426e-01 ... -1.5543641e+00\n",
      "   -1.0761261e+00 -3.3174269e+00]\n",
      "  [-1.6459581e+00 -1.0430148e+00  7.2840986e+00 ...  1.1135731e+00\n",
      "   -2.6066999e+00 -1.0346651e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[-1.6086155e+00 -1.4523654e+00 -9.7418435e-02 ...  2.0421238e+00\n",
      "    4.9691209e-01  2.4465302e-01]\n",
      "  [-2.0852821e+00  2.4312077e+00 -3.0380690e+00 ... -1.7337207e+00\n",
      "    2.7863145e+00 -3.6222312e+00]\n",
      "  [-2.8763404e+00 -3.0851215e-01 -6.8313980e-01 ...  1.9171695e+00\n",
      "    1.4589757e-01 -8.3710933e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[-1.5954803e+00  2.4087257e+00  3.6327033e+00 ... -4.1610655e-01\n",
      "   -8.3287263e-01 -2.1594110e+00]\n",
      "  [-2.7618613e+00  7.8031230e+00  1.1297382e-01 ... -1.3096650e+00\n",
      "   -1.1511629e+00 -3.1324937e+00]\n",
      "  [-1.6010338e+00 -4.7008172e-02  6.7832117e+00 ...  1.0408899e+00\n",
      "   -2.6779552e+00 -1.4747190e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 5.3762525e-01 -1.5987177e+00 -2.3755906e+00 ...  1.0237819e-01\n",
      "    1.4107717e+00  6.5917724e-01]\n",
      "  [ 7.9415331e+00 -1.4741528e+00 -1.2896566e+00 ... -9.5806909e-01\n",
      "   -1.9704407e+00 -2.5434203e+00]\n",
      "  [ 8.0455303e+00 -1.1602191e+00 -1.7030180e+00 ... -1.2803310e+00\n",
      "   -1.4844764e+00 -2.8317347e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[-2.8753340e-01 -1.5651715e+00 -1.6061707e+00 ...  2.4962714e+00\n",
      "    1.0353978e-01 -1.2069447e+00]\n",
      "  [ 2.0707017e-01  2.8764474e-01 -3.0703340e+00 ... -5.2721626e-01\n",
      "    1.5716063e+00 -2.5825093e+00]\n",
      "  [ 3.2499993e-01 -2.8674436e+00  4.3928540e-01 ...  4.3606730e+00\n",
      "   -2.1487408e+00  1.0058823e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n",
      "F1-score of [de-fr] model on [en] dataset: 0.668\n"
     ]
    }
   ],
   "source": [
    "\n",
    "for lang in languages:\n",
    "    f1 = evaluate_lang_performance(lang, trainer)\n",
    "    print(f\"F1-score of [de-fr] model on [{lang}] dataset: {f1:.3f}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Comparing the performance of fine-tuning on each language separately against multilingual learning on all the corpora.\n",
    "\n",
    "\n",
    "Since we have already fine-tuned on the German corpus, we can fine-tune on\n",
    "the remaining languages with our **`fine_tuning_training_on_single_corpus()`** function, with num_samples equal to the number of examples in the training set:\n",
    "\n",
    "#### Note this `fine_tuning_training_on_single_corpus()` has the full training code for a single dataset. So it will take quite a bit of time to run the below cell as Full Training will be executed on each of the different language-corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "3AhZDqN13pUy",
    "outputId": "87b5716b-83fa-4b91-b1b1-84f571d67294"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.fr/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-92119c01382f718a.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.fr/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-df08c9bf8a078f41.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.fr/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-5a8067996ffe172e.arrow\n",
      "Loading cached shuffled indices for dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.fr/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-80d9e756466ec551.arrow\n",
      "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:310: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  FutureWarning,\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='573' max='573' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [573/573 02:47, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.547100</td>\n",
       "      <td>0.318010</td>\n",
       "      <td>0.774057</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.262900</td>\n",
       "      <td>0.287602</td>\n",
       "      <td>0.826218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.180400</td>\n",
       "      <td>0.272368</td>\n",
       "      <td>0.841891</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 5.0425255e-01 -1.3537556e-01 -6.2853122e-01 ... -5.2712657e-03\n",
      "   -6.5075189e-01  2.7888361e-01]\n",
      "  [-1.4131128e+00  5.6016569e+00 -4.5227185e-01 ... -1.7317444e+00\n",
      "    1.5323001e-01 -3.1015322e+00]\n",
      "  [-1.1516833e+00  2.6841943e+00  3.3921769e+00 ... -3.2072619e-01\n",
      "   -2.1890142e+00 -1.2911124e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.4210643e-01 -4.0894052e-01 -7.2958332e-01 ...  2.9312393e-01\n",
      "   -6.0974610e-01  6.1789179e-01]\n",
      "  [ 5.9397902e+00 -1.1361338e+00 -9.9149245e-01 ... -1.1154945e+00\n",
      "   -4.7490063e-01 -2.1339757e+00]\n",
      "  [ 5.4439330e+00 -1.7030106e+00 -7.8766102e-01 ... -9.1369063e-01\n",
      "   -7.0414370e-01 -1.6895144e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 5.0352967e-01 -1.8727170e-01 -7.9400277e-01 ...  5.1735330e-01\n",
      "   -7.9031581e-01  2.0164958e-01]\n",
      "  [ 6.5361290e+00 -1.1122607e+00 -1.3208845e+00 ... -9.4114673e-01\n",
      "   -1.7449892e+00 -2.2695045e+00]\n",
      "  [ 6.2387013e+00 -1.0861852e+00 -1.2590815e+00 ... -7.8865123e-01\n",
      "   -1.7510638e+00 -2.3301814e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 4.1369510e-01 -3.3204058e-01 -8.4698939e-01 ...  3.4641373e-01\n",
      "   -7.0092869e-01  3.4783652e-01]\n",
      "  [ 5.6747713e+00 -1.5943698e+00 -1.5149709e+00 ... -6.2917793e-01\n",
      "   -1.6365278e+00 -2.6905975e+00]\n",
      "  [ 4.9400883e+00 -2.0086689e+00 -1.0519189e+00 ...  6.2359864e-01\n",
      "   -2.2706287e+00 -1.7352808e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.2885715e-01 -3.3493271e-01 -7.1216965e-01 ...  1.5673128e-01\n",
      "   -5.0582552e-01  6.5573847e-01]\n",
      "  [ 6.1795607e+00 -1.2629800e+00 -1.1124971e+00 ... -1.2197809e+00\n",
      "   -7.1382427e-01 -2.1076128e+00]\n",
      "  [ 5.4663310e+00 -1.9121126e+00 -7.7854985e-01 ... -1.0069234e+00\n",
      "   -9.0684104e-01 -1.5189575e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.9655104e-01 -2.3085527e-01 -7.7334309e-01 ...  2.1065181e-01\n",
      "   -5.8826935e-01  3.5187116e-01]\n",
      "  [-2.9123315e-01  1.2775255e+00 -3.0193105e+00 ... -1.1189713e+00\n",
      "    1.1672230e+00 -3.5907564e+00]\n",
      "  [-4.1496024e-01 -1.2297628e+00 -3.0740562e-01 ...  2.8983700e+00\n",
      "   -1.5820307e+00 -1.2903399e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n",
      "eval_pred.predictions  [[[ 6.14817917e-01 -1.58817768e-01 -6.70567632e-01 ... -3.20551507e-02\n",
      "   -6.64358318e-01  1.87191173e-01]\n",
      "  [-1.33130169e+00  5.94492245e+00 -9.87843990e-01 ... -2.11039853e+00\n",
      "    4.52768207e-02 -3.36093807e+00]\n",
      "  [-9.13076758e-01  2.57110715e+00  3.41038060e+00 ... -3.19331229e-01\n",
      "   -2.99998236e+00 -1.19010437e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[-1.38071209e-01 -1.27338064e+00 -1.22888005e+00 ...  2.33598495e+00\n",
      "   -1.03016806e+00  1.02608597e+00]\n",
      "  [ 6.83220100e+00 -1.52106273e+00 -1.47631741e+00 ... -1.49077845e+00\n",
      "   -9.99385893e-01 -2.07586837e+00]\n",
      "  [ 6.56025743e+00 -1.93364727e+00 -1.30461299e+00 ... -1.27616692e+00\n",
      "   -1.26101685e+00 -1.68481863e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 1.13070714e+00 -5.37101924e-01 -1.43514097e+00 ...  1.52872157e+00\n",
      "   -1.11111557e+00 -5.25331616e-01]\n",
      "  [ 6.80219030e+00 -1.75314093e+00 -1.28032303e+00 ... -6.01160526e-01\n",
      "   -1.81535351e+00 -1.92296362e+00]\n",
      "  [ 6.48497677e+00 -1.63395548e+00 -1.46907818e+00 ... -3.74491513e-01\n",
      "   -1.84155738e+00 -1.87989783e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 5.56080282e-01 -7.16694295e-01 -1.32959616e+00 ...  9.77004111e-01\n",
      "   -9.02572155e-01  9.71991122e-02]\n",
      "  [ 6.99633884e+00 -1.93673682e+00 -1.50377536e+00 ... -6.76351428e-01\n",
      "   -1.71950948e+00 -1.86584413e+00]\n",
      "  [ 6.84626102e+00 -1.95498836e+00 -1.36235631e+00 ... -6.03728831e-01\n",
      "   -2.07523417e+00 -1.62811291e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 5.04735529e-01 -8.57896566e-01 -1.16758919e+00 ...  1.09711640e-01\n",
      "    1.20904155e-01  1.40426552e+00]\n",
      "  [ 6.73561096e+00 -1.59702802e+00 -1.48337519e+00 ... -1.60143280e+00\n",
      "   -1.00254989e+00 -2.05775976e+00]\n",
      "  [ 6.40226555e+00 -2.04490542e+00 -1.21675336e+00 ... -1.36480224e+00\n",
      "   -1.21747637e+00 -1.57668555e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 4.34539825e-01 -4.73739117e-01 -9.76404727e-01 ...  5.79986870e-01\n",
      "   -6.25771284e-01  3.16450655e-01]\n",
      "  [ 2.09391825e-02  2.55597770e-01 -3.33806825e+00 ... -1.25891209e+00\n",
      "    1.30532598e+00 -3.30413413e+00]\n",
      "  [ 1.10683672e-01 -2.66904759e+00 -1.42668653e+00 ...  3.61176801e+00\n",
      "   -1.22080719e+00 -4.87548769e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]]\n",
      "eval_pred.predictions  [[[ 6.70152485e-01 -1.90599218e-01 -6.22062385e-01 ... -8.79607797e-02\n",
      "   -6.84798002e-01  1.58532381e-01]\n",
      "  [-1.61105967e+00  6.41241550e+00 -1.08317351e+00 ... -2.04411149e+00\n",
      "   -8.73944089e-02 -3.22386050e+00]\n",
      "  [-1.11639154e+00  2.33755255e+00  3.87791729e+00 ... -1.43447787e-01\n",
      "   -3.19392061e+00 -1.04390430e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[-9.09943402e-01 -1.91943026e+00 -9.15957630e-01 ...  3.64663458e+00\n",
      "   -1.29457927e+00  1.64905059e+00]\n",
      "  [ 7.16543627e+00 -1.57241988e+00 -1.36696947e+00 ... -1.48292994e+00\n",
      "   -1.27702129e+00 -1.93584800e+00]\n",
      "  [ 6.81613159e+00 -1.93648505e+00 -1.23447347e+00 ... -1.27869701e+00\n",
      "   -1.46267140e+00 -1.55638623e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 7.89325893e-01 -3.63517016e-01 -9.56100464e-01 ...  1.99673378e+00\n",
      "   -1.30095923e+00 -9.35661316e-01]\n",
      "  [ 7.00524378e+00 -1.58628666e+00 -1.22112322e+00 ... -8.97822559e-01\n",
      "   -1.94485760e+00 -1.92113221e+00]\n",
      "  [ 6.61093330e+00 -1.63131416e+00 -1.14190638e+00 ... -5.55011094e-01\n",
      "   -1.98359585e+00 -1.92111707e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 9.66474414e-02 -9.99502659e-01 -1.39021647e+00 ...  2.00703335e+00\n",
      "   -1.25857437e+00 -1.10908456e-01]\n",
      "  [ 7.11628389e+00 -1.83787596e+00 -1.27452171e+00 ... -6.95245564e-01\n",
      "   -1.88137448e+00 -1.85226226e+00]\n",
      "  [ 6.92936468e+00 -1.84164310e+00 -1.14294600e+00 ... -6.29324257e-01\n",
      "   -2.20241427e+00 -1.64676464e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 5.09546399e-02 -1.58229899e+00 -1.17229676e+00 ...  3.83926779e-01\n",
      "    8.35572660e-01  2.87850809e+00]\n",
      "  [ 7.08773184e+00 -1.65675271e+00 -1.31434715e+00 ... -1.58264232e+00\n",
      "   -1.36017287e+00 -1.85423291e+00]\n",
      "  [ 6.62857723e+00 -2.01809597e+00 -1.17009926e+00 ... -1.35696089e+00\n",
      "   -1.48273170e+00 -1.43419349e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 3.21118683e-01 -5.53171217e-01 -1.00883818e+00 ...  9.33820486e-01\n",
      "   -7.56624281e-01  2.46374294e-01]\n",
      "  [-3.83068711e-01  4.71309662e-01 -3.19529819e+00 ... -1.30132890e+00\n",
      "    1.04257524e+00 -3.40091443e+00]\n",
      "  [-4.11551982e-01 -2.52399611e+00 -1.09668005e+00 ...  3.88341570e+00\n",
      "   -1.38975835e+00 -4.53411072e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]]\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 1.15762842e+00 -1.47984065e-02 -8.04894447e-01 ... -1.68146506e-01\n",
      "   -7.54665613e-01 -1.50814116e-01]\n",
      "  [ 7.22217941e+00 -1.42589951e+00 -1.24908090e+00 ... -1.30231833e+00\n",
      "   -1.86545157e+00 -2.09823108e+00]\n",
      "  [ 7.20115137e+00 -1.20987892e+00 -1.09086084e+00 ... -1.34134877e+00\n",
      "   -1.76341057e+00 -2.26010084e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 2.20875096e+00 -5.35217047e-01 -1.91191149e+00 ... -1.09106934e+00\n",
      "    1.01995599e+00 -2.68468469e-01]\n",
      "  [ 6.82000113e+00 -1.15437984e+00 -1.55306768e+00 ... -1.81308699e+00\n",
      "   -7.49831557e-01 -2.61019373e+00]\n",
      "  [ 6.96833181e+00 -1.90371180e+00 -9.87802446e-01 ... -1.31672275e+00\n",
      "   -1.35179746e+00 -1.84793234e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 1.21221471e+00 -4.31271829e-02 -6.13241315e-01 ... -2.07118064e-01\n",
      "   -8.79059613e-01 -1.31023288e-01]\n",
      "  [ 7.13019085e+00 -1.25787210e+00 -1.07645416e+00 ... -1.20297730e+00\n",
      "   -2.10128403e+00 -1.96084011e+00]\n",
      "  [ 6.85506201e+00 -1.02606189e+00 -1.18483615e+00 ... -1.34757590e+00\n",
      "   -1.87420082e+00 -1.81305921e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 1.16578504e-01 -7.78424621e-01 -1.29895949e+00 ...  1.41779733e+00\n",
      "   -7.80321240e-01  6.47113562e-01]\n",
      "  [ 6.83337355e+00 -1.57559931e+00 -1.46471655e+00 ... -9.73310411e-01\n",
      "   -1.84131777e+00 -1.79930198e+00]\n",
      "  [-1.21594810e+00  2.70152479e-01 -3.42672586e+00 ... -1.41472840e+00\n",
      "    2.22233200e+00 -2.86693573e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 6.25510454e-01 -1.44912088e+00 -1.21984792e+00 ...  2.66442442e+00\n",
      "   -1.21011376e+00 -4.27025497e-01]\n",
      "  [-1.23601055e+00  8.77408624e-01 -2.54093933e+00 ... -1.27301955e+00\n",
      "    1.07590747e+00 -3.52715349e+00]\n",
      "  [-5.48452675e-01 -3.40592313e+00 -9.23270047e-01 ...  4.21884298e+00\n",
      "   -1.96044779e+00 -1.89891636e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 9.56461489e-01 -1.48427673e-02 -6.34867430e-01 ... -1.31585613e-01\n",
      "   -8.35746646e-01 -9.12322924e-02]\n",
      "  [ 7.22972298e+00 -1.43975484e+00 -1.07192719e+00 ... -1.27216101e+00\n",
      "   -1.86425579e+00 -2.21647239e+00]\n",
      "  [ 7.12756968e+00 -1.21125484e+00 -8.62253964e-01 ... -1.27092659e+00\n",
      "   -1.89461243e+00 -2.34128761e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.it/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-d1da61ce8ecf686a.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.it/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-e071a6993a6540a5.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.it/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-dc3ad1640cb4e180.arrow\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='210' max='210' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [210/210 01:03, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.709300</td>\n",
       "      <td>0.332487</td>\n",
       "      <td>0.741068</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.291800</td>\n",
       "      <td>0.268984</td>\n",
       "      <td>0.796907</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.180100</td>\n",
       "      <td>0.247502</td>\n",
       "      <td>0.822857</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 2.80071139e-01  1.15186885e-01 -9.79367495e-01 ... -1.94526643e-01\n",
      "    7.28902221e-01  4.21614021e-01]\n",
      "  [ 5.98566866e+00 -1.77139831e+00 -1.48365712e+00 ... -6.13943458e-01\n",
      "   -5.79200163e-02 -1.37588930e+00]\n",
      "  [ 6.06399775e+00 -1.82995462e+00 -1.33918858e+00 ... -4.63707566e-01\n",
      "   -5.05947351e-01 -1.09453630e+00]\n",
      "  ...\n",
      "  [ 2.50760406e-01 -2.66857177e-01 -8.60749006e-01 ... -4.09304142e-01\n",
      "    7.37210631e-01  2.44390905e-01]\n",
      "  [ 2.50760406e-01 -2.66857177e-01 -8.60749006e-01 ... -4.09304142e-01\n",
      "    7.37210631e-01  2.44390905e-01]\n",
      "  [ 2.50760406e-01 -2.66857177e-01 -8.60749006e-01 ... -4.09304142e-01\n",
      "    7.37210631e-01  2.44390905e-01]]\n",
      "\n",
      " [[ 1.45614386e+00 -1.20291531e-01 -1.01672781e+00 ... -1.23703308e-01\n",
      "   -4.19911355e-01 -2.72732466e-01]\n",
      "  [ 6.30156946e+00 -1.62766767e+00 -1.22171986e+00 ... -6.31114244e-01\n",
      "   -6.06474519e-01 -1.35395098e+00]\n",
      "  [ 7.15667248e+00 -2.25324821e+00 -1.40170801e+00 ... -7.49725759e-01\n",
      "   -1.04408538e+00 -1.20708025e+00]\n",
      "  ...\n",
      "  [ 6.65996885e+00 -1.72205353e+00 -9.77722287e-01 ... -7.31645465e-01\n",
      "   -1.30269635e+00 -1.17569411e+00]\n",
      "  [ 6.15180683e+00 -1.59053910e+00 -1.41469657e+00 ... -3.15080553e-01\n",
      "   -1.10777867e+00 -1.02724576e+00]\n",
      "  [ 1.51175904e+00 -3.08481097e-01 -1.00467718e+00 ... -2.50208974e-01\n",
      "   -4.30874258e-01 -3.80706191e-01]]\n",
      "\n",
      " [[-1.43072084e-01  1.12782550e+00 -5.50043046e-01 ... -2.26771105e-02\n",
      "   -2.67220169e-01 -9.39831555e-01]\n",
      "  [ 5.41163111e+00 -1.09798455e+00 -7.17212796e-01 ... -8.43094051e-01\n",
      "   -8.47298980e-01 -1.37895238e+00]\n",
      "  [ 4.89694262e+00 -1.02260005e+00 -5.62444508e-01 ... -6.29026830e-01\n",
      "   -6.84989452e-01 -1.29546440e+00]\n",
      "  ...\n",
      "  [ 1.31393170e+00 -6.74337685e-01 -8.23803127e-01 ... -5.41545935e-02\n",
      "   -5.83226979e-01 -2.20278382e-01]\n",
      "  [ 1.31393170e+00 -6.74337685e-01 -8.23803127e-01 ... -5.41545935e-02\n",
      "   -5.83226979e-01 -2.20278382e-01]\n",
      "  [ 1.31393170e+00 -6.74337685e-01 -8.23803127e-01 ... -5.41545935e-02\n",
      "   -5.83226979e-01 -2.20278382e-01]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 1.57257214e-01  8.24339688e-01 -3.58832985e-01 ... -9.57624391e-02\n",
      "   -7.47088790e-01 -1.29417345e-01]\n",
      "  [-9.90045428e-01  5.67303371e+00 -3.54314625e-01 ... -1.50101864e+00\n",
      "   -8.45857382e-01 -1.90874028e+00]\n",
      "  [-1.83860373e+00 -3.19868356e-01  5.20005608e+00 ...  3.70053798e-01\n",
      "   -1.15729046e+00 -5.94615161e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 3.02845448e-01 -5.00606447e-02 -8.42990398e-01 ...  2.38558594e-02\n",
      "   -4.11479831e-01  4.61658627e-01]\n",
      "  [-1.12704611e+00  2.02923298e+00 -6.40455842e-01 ... -1.08326125e+00\n",
      "    1.80653358e+00 -1.23375165e+00]\n",
      "  [-1.93436730e+00  1.21433532e+00  6.85716510e-01 ...  3.86710055e-02\n",
      "    1.29933119e+00 -4.44277674e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 2.75712460e-01 -1.11351088e-01 -9.28876281e-01 ... -6.12458326e-02\n",
      "   -2.29015455e-01  7.00481415e-01]\n",
      "  [-1.09300578e+00 -5.68882376e-02 -6.86227322e-01 ... -7.24922538e-01\n",
      "    2.67970610e+00  4.01216835e-01]\n",
      "  [-1.46880150e+00 -1.87716782e+00  4.42406684e-01 ...  9.83850479e-01\n",
      "    1.06699705e+00  3.66149354e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]]\n",
      "eval_pred.predictions  [[[ 5.68980634e-01 -1.73168406e-01 -1.00119865e+00 ... -1.38213858e-01\n",
      "   -1.40798584e-01  5.76182306e-01]\n",
      "  [ 4.88786602e+00 -1.50736773e+00 -1.45145833e+00 ... -9.81449723e-01\n",
      "    4.99252558e-01 -1.61245239e+00]\n",
      "  [ 5.19824648e+00 -1.77961195e+00 -1.40924597e+00 ... -3.00875008e-01\n",
      "   -4.80596453e-01 -8.97456706e-01]\n",
      "  ...\n",
      "  [ 5.84295213e-01 -3.29871058e-01 -9.89866555e-01 ... -2.25879133e-01\n",
      "   -1.61412314e-01  5.26668191e-01]\n",
      "  [ 5.84295213e-01 -3.29871058e-01 -9.89866555e-01 ... -2.25879133e-01\n",
      "   -1.61412314e-01  5.26668191e-01]\n",
      "  [ 5.84295213e-01 -3.29871058e-01 -9.89866555e-01 ... -2.25879133e-01\n",
      "   -1.61412314e-01  5.26668191e-01]]\n",
      "\n",
      " [[ 8.06742013e-01 -1.25409395e-01 -7.97295034e-01 ... -1.12950571e-01\n",
      "   -5.32669365e-01  1.22223005e-01]\n",
      "  [ 6.76734161e+00 -1.68578887e+00 -8.41246367e-01 ... -1.02248394e+00\n",
      "   -8.19519401e-01 -1.57986295e+00]\n",
      "  [ 6.72783184e+00 -2.10249138e+00 -1.16442788e+00 ... -8.27019334e-01\n",
      "   -1.37676644e+00 -1.03869545e+00]\n",
      "  ...\n",
      "  [ 6.73070765e+00 -1.80623221e+00 -4.59742904e-01 ... -1.05786788e+00\n",
      "   -1.64916146e+00 -1.22589111e+00]\n",
      "  [ 5.53533411e+00 -1.42200315e+00 -9.71822798e-01 ... -5.23485303e-01\n",
      "   -1.37766874e+00 -9.32026565e-01]\n",
      "  [ 8.16260993e-01 -1.91411674e-01 -7.94637620e-01 ... -1.57814845e-01\n",
      "   -5.43607354e-01  9.70528424e-02]]\n",
      "\n",
      " [[ 5.20487309e-01  1.89896747e-01 -8.71041477e-01 ... -1.55125022e-01\n",
      "   -5.71361423e-01 -3.16399932e-02]\n",
      "  [ 6.97608519e+00 -1.40775311e+00 -8.52085114e-01 ... -1.41661429e+00\n",
      "   -1.40985632e+00 -1.64983881e+00]\n",
      "  [ 6.81101799e+00 -1.40368938e+00 -7.66279280e-01 ... -1.27130651e+00\n",
      "   -1.36368608e+00 -1.69628441e+00]\n",
      "  ...\n",
      "  [ 1.29139161e+00 -6.14642799e-01 -8.89776051e-01 ... -2.27901250e-01\n",
      "   -5.88364780e-01 -3.77608746e-01]\n",
      "  [ 1.29139161e+00 -6.14642799e-01 -8.89776051e-01 ... -2.27901250e-01\n",
      "   -5.88364780e-01 -3.77608746e-01]\n",
      "  [ 1.29139161e+00 -6.14642799e-01 -8.89776051e-01 ... -2.27901250e-01\n",
      "   -5.88364780e-01 -3.77608746e-01]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 4.51415747e-01  1.90432996e-01 -6.02067292e-01 ... -1.05961911e-01\n",
      "   -6.59259439e-01  1.58429667e-01]\n",
      "  [-1.21087968e+00  6.15677738e+00 -4.26986307e-01 ... -1.89002991e+00\n",
      "   -8.72720361e-01 -2.27807426e+00]\n",
      "  [-1.90662885e+00 -3.59936118e-01  5.37029982e+00 ...  4.78597224e-01\n",
      "   -1.24058211e+00 -5.81188321e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 3.51377219e-01 -1.01187639e-01 -7.90122092e-01 ...  1.04151137e-01\n",
      "   -5.04148722e-01  3.98098379e-01]\n",
      "  [-1.15656996e+00  5.06178737e-01 -1.93128538e+00 ... -5.48665702e-01\n",
      "    2.09604692e+00 -1.78904986e+00]\n",
      "  [-1.61649823e+00 -9.30332124e-01 -6.58226192e-01 ...  2.19910526e+00\n",
      "    7.86239207e-01  1.04019061e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 4.50377226e-01 -1.84050664e-01 -8.72876167e-01 ... -9.00398269e-02\n",
      "   -3.54298890e-01  6.17607355e-01]\n",
      "  [-8.33259463e-01 -3.16605121e-01 -1.58302712e+00 ... -1.30508053e+00\n",
      "    3.60554433e+00  1.19349390e-01]\n",
      "  [-9.23560083e-01 -2.34208250e+00 -4.79898751e-01 ...  1.23296583e+00\n",
      "    5.29447854e-01  4.60221434e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]]\n",
      "eval_pred.predictions  [[[ 8.7533414e-01 -6.6732720e-02 -1.0528417e+00 ... -3.9759520e-01\n",
      "   -8.3794698e-02  4.5643294e-01]\n",
      "  [ 6.5290232e+00 -1.3998133e+00 -1.1803527e+00 ... -1.4452567e+00\n",
      "   -2.3614331e-01 -1.9193134e+00]\n",
      "  [ 6.6238041e+00 -1.7020148e+00 -1.2282124e+00 ... -9.4975305e-01\n",
      "   -8.1052524e-01 -1.4642485e+00]\n",
      "  ...\n",
      "  [ 8.7448716e-01 -2.5589350e-01 -1.0256100e+00 ... -4.8904875e-01\n",
      "   -1.3115598e-01  3.9091754e-01]\n",
      "  [ 8.7448716e-01 -2.5589350e-01 -1.0256100e+00 ... -4.8904875e-01\n",
      "   -1.3115598e-01  3.9091754e-01]\n",
      "  [ 8.7448716e-01 -2.5589350e-01 -1.0256100e+00 ... -4.8904875e-01\n",
      "   -1.3115598e-01  3.9091754e-01]]\n",
      "\n",
      " [[ 9.9379849e-01 -8.6568661e-02 -8.1341630e-01 ... -2.0747848e-01\n",
      "   -5.5770493e-01  1.0689047e-02]\n",
      "  [ 7.0474930e+00 -1.5328730e+00 -8.1124085e-01 ... -1.2567385e+00\n",
      "   -9.4507557e-01 -1.7833512e+00]\n",
      "  [ 7.3704047e+00 -2.0145023e+00 -1.1504929e+00 ... -1.1354399e+00\n",
      "   -1.4719431e+00 -1.3686690e+00]\n",
      "  ...\n",
      "  [ 7.0451908e+00 -1.6239477e+00 -5.8902544e-01 ... -1.2762042e+00\n",
      "   -1.6945510e+00 -1.5071669e+00]\n",
      "  [ 6.1484299e+00 -1.3325683e+00 -1.0691441e+00 ... -8.2377750e-01\n",
      "   -1.5205616e+00 -1.2679917e+00]\n",
      "  [ 1.0004824e+00 -1.5863994e-01 -8.0887020e-01 ... -2.5715092e-01\n",
      "   -5.7023871e-01 -1.4114919e-02]]\n",
      "\n",
      " [[ 5.8555293e-01  3.0181319e-01 -8.6762649e-01 ... -2.3177285e-01\n",
      "   -5.8235753e-01 -1.3404581e-01]\n",
      "  [ 7.1610212e+00 -1.0895807e+00 -8.8623852e-01 ... -1.5234863e+00\n",
      "   -1.4621955e+00 -1.9135871e+00]\n",
      "  [ 6.9479485e+00 -1.0464057e+00 -7.7416676e-01 ... -1.4208895e+00\n",
      "   -1.4040540e+00 -2.0035098e+00]\n",
      "  ...\n",
      "  [ 1.4163324e+00 -5.0195450e-01 -9.0593421e-01 ... -3.6599270e-01\n",
      "   -6.3907057e-01 -5.0716388e-01]\n",
      "  [ 1.4163324e+00 -5.0195450e-01 -9.0593421e-01 ... -3.6599270e-01\n",
      "   -6.3907057e-01 -5.0716388e-01]\n",
      "  [ 1.4163324e+00 -5.0195450e-01 -9.0593421e-01 ... -3.6599270e-01\n",
      "   -6.3907057e-01 -5.0716388e-01]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 4.7354853e-01  2.4854890e-01 -5.8611095e-01 ... -1.3892870e-01\n",
      "   -6.7377174e-01  1.1449155e-01]\n",
      "  [-9.8707038e-01  6.2998919e+00 -4.2280114e-01 ... -1.9736472e+00\n",
      "   -1.1251900e+00 -2.2893107e+00]\n",
      "  [-2.0293169e+00 -4.6980968e-01  5.9128838e+00 ...  5.1097190e-01\n",
      "   -1.6136011e+00 -6.5290594e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.4292471e-01 -5.5508753e-03 -6.9917583e-01 ...  2.8036799e-02\n",
      "   -5.3248829e-01  3.3247933e-01]\n",
      "  [-1.0426289e+00  2.6299953e+00 -1.2627070e+00 ... -1.5622207e+00\n",
      "    1.5724033e+00 -2.4198372e+00]\n",
      "  [-2.0003531e+00  1.1086888e+00  6.8476444e-01 ...  9.2928582e-01\n",
      "    4.6656209e-01 -1.2613226e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 5.0223166e-01 -1.6412732e-01 -8.9342934e-01 ... -1.7561947e-01\n",
      "   -3.1942129e-01  6.3483000e-01]\n",
      "  [-3.9544189e-01 -3.3082247e-01 -1.8925152e+00 ... -1.8822773e+00\n",
      "    4.1729503e+00  2.6960737e-01]\n",
      "  [-6.1111772e-01 -2.2954049e+00 -3.9583731e-01 ...  5.8797443e-01\n",
      "    6.0049576e-01  5.1470714e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 7.8188926e-01 -7.6823026e-02 -8.5669750e-01 ... -1.3643132e-01\n",
      "   -5.0978035e-01  1.4746037e-01]\n",
      "  [ 7.2378178e+00 -1.9355386e+00 -6.2783802e-01 ... -1.7007271e+00\n",
      "   -9.0517962e-01 -1.7615310e+00]\n",
      "  [ 7.0137644e+00 -1.7634940e+00 -1.1340739e+00 ... -1.5350876e+00\n",
      "   -7.0575738e-01 -1.6335953e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 8.9902860e-01 -4.1968107e-02 -9.6865600e-01 ... -2.7319741e-01\n",
      "   -3.1628236e-01  2.6962712e-01]\n",
      "  [ 7.1221099e+00 -1.6336442e+00 -9.4876200e-01 ... -1.5766616e+00\n",
      "   -9.7348374e-01 -1.9714366e+00]\n",
      "  [ 7.4429326e+00 -1.9991225e+00 -1.0359839e+00 ... -1.3084871e+00\n",
      "   -1.5491306e+00 -1.4602226e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.9837471e-01 -8.5023403e-02 -8.8068908e-01 ...  1.6026680e-01\n",
      "   -5.6069666e-01  2.6162621e-01]\n",
      "  [-1.0258613e+00  1.5756744e-01 -2.1856561e+00 ...  1.7361292e-01\n",
      "    4.7472405e-01 -2.5743659e+00]\n",
      "  [-1.3938247e+00 -2.1561148e+00 -8.8343960e-01 ...  5.3569160e+00\n",
      "   -1.6263621e+00  4.1811289e-03]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 3.4294432e-01  5.2263039e-01 -4.9756554e-01 ... -1.4130786e-01\n",
      "   -7.4497420e-01  3.2975551e-02]\n",
      "  [ 6.1562905e+00 -9.8511136e-01 -1.5667235e+00 ... -1.1348165e+00\n",
      "   -1.0015711e+00 -2.3653691e+00]\n",
      "  [ 5.9616671e+00 -1.4695354e+00 -1.0512674e+00 ... -8.2268709e-01\n",
      "   -1.5181221e+00 -1.9718270e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 2.5999179e-01  6.5430790e-02 -8.5307884e-01 ...  3.9228636e-01\n",
      "   -6.6659850e-01  7.4002832e-02]\n",
      "  [-3.3448142e-01  4.5048612e-01 -2.0080202e+00 ... -6.4373767e-01\n",
      "    4.6415842e-01 -2.8911986e+00]\n",
      "  [-6.4565301e-01 -1.1666685e+00 -1.0708576e+00 ...  2.8024058e+00\n",
      "   -9.1817123e-01 -2.1050925e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.6909972e-01 -5.7036698e-02 -8.9228845e-01 ...  3.9717129e-01\n",
      "   -7.0170975e-01  1.6251734e-01]\n",
      "  [-1.3657663e+00 -2.1140113e-01 -2.1429098e+00 ...  6.3872087e-01\n",
      "    2.1733859e-01 -2.5929167e+00]\n",
      "  [-6.5801609e-01 -2.4320729e+00  1.3847789e-01 ...  5.1666303e+00\n",
      "   -2.2415302e+00 -1.9850316e-02]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.en/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-9c7d0d5777a90be6.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.en/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-8f6662f67966a072.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/xtreme/PAN-X.en/1.0.0/29f5d57a48779f37ccb75cb8708d1095448aad0713b425bdc1ff9a4a128a56e4/cache-bb9f36423604015e.arrow\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='150' max='150' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [150/150 00:44, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1.022300</td>\n",
       "      <td>0.498726</td>\n",
       "      <td>0.577873</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.493900</td>\n",
       "      <td>0.399291</td>\n",
       "      <td>0.660773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.362300</td>\n",
       "      <td>0.373787</td>\n",
       "      <td>0.697207</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 3.1733844e-01 -1.7696494e-01 -7.9041970e-01 ...  2.2110416e-01\n",
      "   -5.9518802e-01  4.4413266e-01]\n",
      "  [ 4.8934770e+00 -1.4914415e+00 -9.4258225e-01 ...  3.2855469e-01\n",
      "   -1.3840483e+00 -1.1328795e+00]\n",
      "  [ 1.2282220e+00 -4.1129351e-01 -8.7591094e-01 ...  1.1821489e+00\n",
      "   -5.3672463e-01 -1.0354943e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.6711147e-01 -2.2652930e-01 -7.6079881e-01 ... -4.3312687e-02\n",
      "   -5.6792396e-01  4.6648088e-01]\n",
      "  [ 6.1253095e+00 -1.4965594e+00 -8.8580310e-01 ... -5.6415892e-01\n",
      "   -1.5338635e+00 -1.1779287e+00]\n",
      "  [ 5.0220962e+00 -1.0574641e+00 -7.0078850e-01 ... -6.7236483e-01\n",
      "   -9.8756522e-01 -1.5546021e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.2311600e-01 -2.7838859e-01 -8.4147447e-01 ... -3.8409930e-02\n",
      "   -4.0670559e-01  6.7204046e-01]\n",
      "  [ 3.7793124e+00 -2.4879456e+00 -1.6203043e+00 ... -1.5844727e-01\n",
      "   -2.6611382e-01  8.8576537e-01]\n",
      "  [-1.9415040e+00 -4.4101673e-01 -1.1218613e+00 ... -1.0750921e+00\n",
      "    3.2269678e+00  1.3164282e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 3.8775352e-01 -2.1730122e-01 -7.8876072e-01 ... -4.1774493e-02\n",
      "   -4.8370251e-01  5.3279912e-01]\n",
      "  [ 5.8599777e+00 -1.9754987e+00 -9.8458898e-01 ...  5.7481084e-02\n",
      "   -1.4215978e+00 -1.0403594e+00]\n",
      "  [-1.7970023e+00 -3.3709818e-01 -1.1849763e+00 ... -1.2853431e+00\n",
      "    3.2529857e+00  7.2431803e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 2.2070710e-01  7.5105548e-02 -5.5697978e-01 ... -8.8644050e-02\n",
      "   -6.6137588e-01  3.7660265e-01]\n",
      "  [-1.5368556e+00  3.7687721e+00  7.2172087e-01 ... -1.8212501e+00\n",
      "    5.9206623e-01 -1.6135559e+00]\n",
      "  [-7.4781251e-01  1.6765052e+00  2.3757379e+00 ... -7.3451376e-01\n",
      "   -5.9477025e-01 -1.1433282e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.0232843e-01 -2.3419344e-01 -7.6342857e-01 ...  7.7174783e-02\n",
      "   -5.8503038e-01  4.3324956e-01]\n",
      "  [ 5.8558426e+00 -1.7949086e+00 -1.0861899e+00 ... -2.0597735e-01\n",
      "   -1.3861929e+00 -1.1642501e+00]\n",
      "  [ 5.3829875e+00 -1.7825736e+00 -8.4279084e-01 ... -2.0648260e-02\n",
      "   -1.1992285e+00 -1.0278754e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n",
      "eval_pred.predictions  [[[ 3.9019048e-01 -2.2191317e-01 -7.6601213e-01 ...  2.1464142e-01\n",
      "   -6.5006673e-01  4.4973794e-01]\n",
      "  [ 5.5497413e+00 -1.6183221e+00 -1.2557448e+00 ... -4.0759161e-01\n",
      "   -1.4834694e+00 -1.2600096e+00]\n",
      "  [ 1.0065614e+00  5.3310301e-02 -1.5245835e+00 ...  8.1167060e-01\n",
      "   -1.9959648e-01 -1.8731127e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 5.3690886e-01 -2.1155101e-01 -7.2387803e-01 ... -5.5489872e-02\n",
      "   -6.1308724e-01  4.5866945e-01]\n",
      "  [ 6.4054661e+00 -1.3683282e+00 -9.3183595e-01 ... -6.7405683e-01\n",
      "   -1.6516535e+00 -1.4724603e+00]\n",
      "  [ 5.4284301e+00 -7.9640269e-01 -1.3116757e+00 ... -1.0425841e+00\n",
      "   -1.0187145e+00 -2.0613508e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.3087810e-01 -3.7352961e-01 -8.5550469e-01 ... -5.8937211e-02\n",
      "   -3.9723954e-01  8.0443579e-01]\n",
      "  [ 5.5138054e+00 -2.0944395e+00 -1.1374967e+00 ... -2.7530572e-01\n",
      "   -1.5862557e+00 -2.6976183e-01]\n",
      "  [-1.5373367e+00 -3.8683385e-01 -1.1640801e+00 ... -1.2600625e+00\n",
      "    3.8892055e+00  6.4978069e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 5.4286939e-01 -2.7450159e-01 -7.7167100e-01 ... -4.2765129e-02\n",
      "   -5.3381145e-01  5.0961185e-01]\n",
      "  [ 6.2402740e+00 -1.7017721e+00 -1.3613986e+00 ... -4.5973489e-01\n",
      "   -1.5288633e+00 -1.1699930e+00]\n",
      "  [-6.6354960e-01 -6.6569634e-02 -2.0105603e+00 ... -1.8424125e+00\n",
      "    3.8542073e+00 -2.9722816e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.1968626e-01 -1.4120796e-02 -5.6945699e-01 ... -5.0925117e-02\n",
      "   -6.8443257e-01  3.7591335e-01]\n",
      "  [-1.2065221e+00  4.5655046e+00 -7.7418929e-01 ... -2.0959122e+00\n",
      "    6.2794584e-01 -2.0973928e+00]\n",
      "  [-3.6823791e-01  1.2517500e+00  2.8355746e+00 ...  1.3292132e-01\n",
      "   -1.8109310e+00 -1.1708552e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.5387679e-01 -2.1486981e-01 -7.4065804e-01 ...  5.5419851e-02\n",
      "   -6.1758977e-01  4.2004928e-01]\n",
      "  [ 6.3091717e+00 -1.6659613e+00 -1.1014439e+00 ... -3.2026479e-01\n",
      "   -1.6415910e+00 -1.3503373e+00]\n",
      "  [ 6.1385655e+00 -1.7484791e+00 -1.1268164e+00 ... -2.4512205e-01\n",
      "   -1.4797314e+00 -1.2732662e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n",
      "eval_pred.predictions  [[[ 3.5221553e-01 -1.8664944e-01 -7.5479770e-01 ...  2.4959455e-01\n",
      "   -6.2545258e-01  4.2216536e-01]\n",
      "  [ 5.3985653e+00 -1.4426097e+00 -1.4537662e+00 ... -3.0527645e-01\n",
      "   -1.3567905e+00 -1.4973996e+00]\n",
      "  [ 2.9692945e-01  6.5313536e-01 -1.6433860e+00 ...  9.7587937e-01\n",
      "   -9.6931852e-02 -2.2851520e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.7558293e-01 -1.7716321e-01 -7.0503950e-01 ... -9.3456887e-02\n",
      "   -5.7404476e-01  4.6102577e-01]\n",
      "  [ 6.5918322e+00 -1.2449307e+00 -1.1522820e+00 ... -7.6525062e-01\n",
      "   -1.6084158e+00 -1.5833706e+00]\n",
      "  [ 5.5220590e+00 -6.8009430e-01 -1.5203974e+00 ... -1.1472152e+00\n",
      "   -9.8065507e-01 -2.0208750e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.9247009e-01 -3.6396584e-01 -8.5033840e-01 ... -9.5807143e-02\n",
      "   -3.5979855e-01  8.0995876e-01]\n",
      "  [ 5.5966344e+00 -1.9814565e+00 -1.2985225e+00 ... -2.6019979e-01\n",
      "   -1.4828838e+00 -5.4299980e-01]\n",
      "  [-1.7084575e+00 -1.3794208e-01 -1.2494806e+00 ... -1.3966597e+00\n",
      "    4.2617583e+00  4.3501547e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 5.0177646e-01 -2.8607345e-01 -7.6750886e-01 ... -7.4996904e-02\n",
      "   -4.9571475e-01  5.6059867e-01]\n",
      "  [ 6.1428971e+00 -1.4964362e+00 -1.5090120e+00 ... -5.2443194e-01\n",
      "   -1.4105990e+00 -1.3855277e+00]\n",
      "  [-1.2178254e+00 -2.1895881e-01 -2.0677421e+00 ... -1.9380542e+00\n",
      "    4.7454867e+00  2.3099762e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 2.6192349e-01  7.0328258e-02 -5.0533390e-01 ... -1.0323297e-01\n",
      "   -6.9089860e-01  3.6129138e-01]\n",
      "  [-1.5393747e+00  5.3556857e+00 -6.6017801e-01 ... -2.2076991e+00\n",
      "    6.1935222e-01 -2.0801828e+00]\n",
      "  [-1.1296859e+00  2.0036721e+00  3.4365141e+00 ... -2.9173058e-01\n",
      "   -1.4950058e+00 -1.1969306e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.3269005e-01 -1.9787690e-01 -7.3441547e-01 ...  5.8121283e-02\n",
      "   -6.0916454e-01  4.0169349e-01]\n",
      "  [ 6.4377794e+00 -1.5407484e+00 -1.3429289e+00 ... -3.4740573e-01\n",
      "   -1.5529821e+00 -1.4990180e+00]\n",
      "  [ 6.2485113e+00 -1.6123142e+00 -1.3235004e+00 ... -1.7660503e-01\n",
      "   -1.4023886e+00 -1.4757096e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 5.6751502e-01 -2.6132488e-01 -7.7602798e-01 ... -1.0058281e-02\n",
      "   -6.0254061e-01  3.9059958e-01]\n",
      "  [ 6.5348849e+00 -1.5539528e+00 -1.6618949e+00 ... -6.6485405e-01\n",
      "   -1.3087538e+00 -1.6130887e+00]\n",
      "  [ 6.5021644e+00 -1.6827965e+00 -1.6360095e+00 ... -5.9766191e-01\n",
      "   -1.4464902e+00 -1.3725111e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.4705952e-01  6.9233678e-02 -5.5396092e-01 ... -1.1049560e-01\n",
      "   -6.9639808e-01  3.3554932e-01]\n",
      "  [-8.0745751e-01  5.0687537e+00 -6.8267834e-01 ... -1.7894850e+00\n",
      "   -4.6550125e-01 -2.4104626e+00]\n",
      "  [-1.2124417e+00  1.0630804e-01  4.8754258e+00 ...  1.0298392e+00\n",
      "   -1.9383799e+00 -8.1971925e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 2.9503104e-01 -2.8307676e-01 -7.7529013e-01 ... -5.2587580e-02\n",
      "   -4.1228712e-01  6.7570168e-01]\n",
      "  [-2.0330689e+00  1.1649343e+00 -1.8933334e+00 ... -1.9061445e+00\n",
      "    4.0685029e+00 -5.9687704e-01]\n",
      "  [-1.9556811e+00 -1.7034957e+00  2.3824988e-02 ...  2.4089298e-01\n",
      "    2.1254168e+00  2.5869672e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 3.1968498e-01  2.3371035e-02 -6.0042882e-01 ... -8.5499227e-02\n",
      "   -6.7058313e-01  3.5228074e-01]\n",
      "  [-9.4117028e-01  4.5969853e+00 -7.0143998e-01 ... -1.8469844e+00\n",
      "   -5.4209143e-01 -2.0637648e+00]\n",
      "  [-6.7251962e-01  1.9165891e-01  4.5747085e+00 ...  1.0430111e+00\n",
      "   -2.2788620e+00 -8.0032068e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.7436989e-01 -2.7730507e-01 -7.7646554e-01 ...  9.7966373e-02\n",
      "   -5.7446158e-01  4.8639089e-01]\n",
      "  [ 6.1362114e+00 -1.6598748e+00 -1.5398279e+00 ... -5.6276888e-01\n",
      "   -1.4123988e+00 -1.1872638e+00]\n",
      "  [ 6.1787391e+00 -1.6335635e+00 -1.6110476e+00 ... -4.8861122e-01\n",
      "   -1.4022431e+00 -1.1135912e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.6496195e-01 -3.0038515e-01 -7.9113162e-01 ...  1.2837647e-01\n",
      "   -5.7167083e-01  4.9240607e-01]\n",
      "  [-1.4621074e+00  9.4380426e-01 -2.2623837e+00 ... -2.9258680e-01\n",
      "    1.7224660e+00 -2.1415701e+00]\n",
      "  [-2.9601511e-01 -3.1374090e+00  5.0357497e-01 ...  4.6936440e+00\n",
      "   -1.4697399e+00  2.0558076e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "corpora = [panx_de_encoded]\n",
    "\n",
    "# Exclude German from iteration\n",
    "for lang in languages[1:]:\n",
    "    training_args.output_dir = f\"xlm-roberta-base-finetuned-panx-{lang}\"\n",
    "    # Fine-tune on monolingual corpus\n",
    "    ds_encoded = encode_panx_dataset(panx_ds_combined[lang])\n",
    "    metrics = fine_tuning_training_on_single_corpus(ds_encoded, ds_encoded[\"train\"].num_rows)\n",
    "    # Collect F1-scores in common dict\n",
    "    f1_scores[lang][lang] = metrics[\"f1_score\"][0]\n",
    "    # Add monolingual corpus to list of corpora to concatenate\n",
    "    corpora.append(ds_encoded)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "id": "uMRUtEdC3pUy"
   },
   "outputs": [],
   "source": [
    "corpora_encoded = concatenate_splits(corpora)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "6Y8rKrfr3pUy",
    "outputId": "f054a2ed-c30b-440a-cd82-b07e534edfe5"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:310: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  FutureWarning,\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='2505' max='2505' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [2505/2505 13:36, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.299200</td>\n",
       "      <td>0.193584</td>\n",
       "      <td>0.816409</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.158800</td>\n",
       "      <td>0.171052</td>\n",
       "      <td>0.846557</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.102200</td>\n",
       "      <td>0.173101</td>\n",
       "      <td>0.852467</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 2.9919024e+00 -7.6689059e-01 -1.4890326e+00 ... -6.7211676e-01\n",
      "   -5.8668792e-01 -6.7394668e-01]\n",
      "  [ 8.0442200e+00 -2.0944490e+00 -2.1246192e+00 ... -5.6890202e-01\n",
      "   -1.9451729e+00 -1.7060621e+00]\n",
      "  [ 7.7958932e+00 -2.2906582e+00 -2.0266480e+00 ... -3.9295155e-01\n",
      "   -1.9943413e+00 -1.3476235e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 2.8786671e+00 -1.2848481e+00 -1.6121919e+00 ... -6.7919368e-01\n",
      "   -1.9804814e-01  3.9747661e-01]\n",
      "  [ 1.6063884e+00  4.7341388e-01 -3.6248176e+00 ... -2.3605833e+00\n",
      "    4.6657639e+00 -1.9360447e+00]\n",
      "  [ 1.3269771e+00 -1.8284042e+00 -1.5502671e+00 ... -3.0514562e-01\n",
      "    1.4326315e+00  1.6327080e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 9.1320843e-01 -3.4928954e-01 -8.6121726e-01 ... -1.8438463e-01\n",
      "   -4.6640229e-01  7.9977304e-02]\n",
      "  [ 6.1559987e+00 -1.8280699e+00 -1.7191379e+00 ... -9.3389654e-01\n",
      "   -1.6201445e+00 -1.3960857e+00]\n",
      "  [ 5.8470879e+00 -1.7258314e+00 -1.5772804e+00 ... -7.7532333e-01\n",
      "   -1.4509670e+00 -1.2208869e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 2.9234262e+00 -1.2501855e+00 -1.7319955e+00 ... -6.3160473e-01\n",
      "   -1.5497729e-01  8.3997518e-02]\n",
      "  [ 8.0411777e+00 -2.3359385e+00 -2.3262942e+00 ... -8.0614114e-01\n",
      "   -1.6920437e+00 -1.1790187e+00]\n",
      "  [ 8.2270632e+00 -2.4638631e+00 -2.3753068e+00 ... -5.9450752e-01\n",
      "   -1.5787770e+00 -1.0593785e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.5443795e+00 -1.1059226e+00 -1.8967276e+00 ... -8.4074461e-01\n",
      "   -3.2049042e-01 -5.5184042e-01]\n",
      "  [ 8.0094490e+00 -1.8423647e+00 -2.4799490e+00 ... -9.1463590e-01\n",
      "   -1.6765456e+00 -1.5538106e+00]\n",
      "  [ 8.0376892e+00 -2.2094500e+00 -2.3162563e+00 ... -3.7287462e-01\n",
      "   -1.9651163e+00 -1.0391243e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 9.7001016e-01 -7.7967644e-01 -1.1789293e+00 ...  3.2484567e-01\n",
      "   -3.8831705e-01  9.5738345e-01]\n",
      "  [-2.4478469e+00  1.5535939e-01 -2.8632691e+00 ... -4.8952585e-01\n",
      "    3.9777994e+00 -6.7725557e-01]\n",
      "  [-1.5367545e+00 -3.1426849e+00  3.9943936e-01 ...  3.5743802e+00\n",
      "   -1.0887845e+00  3.9997418e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n",
      "eval_pred.predictions  [[[ 4.9729590e+00 -1.1521926e+00 -1.4275688e+00 ... -1.1336857e+00\n",
      "   -9.7835857e-01 -2.2220435e+00]\n",
      "  [ 8.8710108e+00 -2.0923939e+00 -1.8282045e+00 ... -8.4348309e-01\n",
      "   -1.9896718e+00 -1.6842310e+00]\n",
      "  [ 8.8516636e+00 -2.2590373e+00 -1.8281736e+00 ... -7.8770632e-01\n",
      "   -2.0080252e+00 -1.5484246e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.4039636e+00 -1.9497358e+00 -1.5667597e+00 ... -7.1540576e-01\n",
      "   -2.9188317e-01  8.9114749e-01]\n",
      "  [-2.7675414e-01  1.6824089e+00 -3.7939131e+00 ... -2.6201358e+00\n",
      "    5.3044500e+00 -2.6938016e+00]\n",
      "  [-5.5680943e-01 -1.9553945e+00 -1.0416989e+00 ...  9.5116854e-02\n",
      "    1.9806659e+00  1.8647681e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 8.5004985e-01 -2.7537832e-01 -8.4036100e-01 ... -2.7029225e-01\n",
      "   -4.2592534e-01 -2.1450041e-02]\n",
      "  [ 8.4164562e+00 -1.5798330e+00 -1.4506565e+00 ... -1.3757792e+00\n",
      "   -2.2988489e+00 -1.9314868e+00]\n",
      "  [ 8.3705444e+00 -1.7443945e+00 -1.2461296e+00 ... -1.1083587e+00\n",
      "   -2.4216523e+00 -1.8412955e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 4.2003837e+00 -2.0920539e+00 -2.2034044e+00 ... -1.3692677e+00\n",
      "    2.1343461e-01 -1.4202812e-01]\n",
      "  [ 9.0288792e+00 -2.1015308e+00 -2.1616123e+00 ... -1.3793676e+00\n",
      "   -1.6651604e+00 -1.3782344e+00]\n",
      "  [ 9.0343657e+00 -2.0958755e+00 -2.2779577e+00 ... -1.3128124e+00\n",
      "   -1.3484102e+00 -1.3674226e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.7978311e+00 -1.8156183e+00 -2.1288810e+00 ... -1.3681061e+00\n",
      "   -2.5560889e-01 -9.3389785e-01]\n",
      "  [ 9.0820408e+00 -1.8282416e+00 -2.0472424e+00 ... -1.3049240e+00\n",
      "   -2.0189898e+00 -1.5021811e+00]\n",
      "  [ 9.0616894e+00 -2.0862992e+00 -1.9522758e+00 ... -8.6034137e-01\n",
      "   -2.1549249e+00 -1.2130907e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[-3.6014792e-01 -7.1524745e-01 -8.5957813e-01 ...  9.6247786e-01\n",
      "   -3.7583914e-01  1.1499625e+00]\n",
      "  [-3.3756657e+00  9.4957322e-01 -2.9262695e+00 ... -1.1400636e+00\n",
      "    4.4337139e+00 -1.7515149e+00]\n",
      "  [-2.4017520e+00 -3.7031448e+00  1.2899195e+00 ...  4.6398416e+00\n",
      "   -1.8467344e+00  4.1834130e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n",
      "eval_pred.predictions  [[[ 4.5103803e+00 -9.3939477e-01 -1.3950677e+00 ... -1.3120276e+00\n",
      "   -8.4937733e-01 -2.0876753e+00]\n",
      "  [ 9.2447557e+00 -1.9949157e+00 -2.1298442e+00 ... -1.3373537e+00\n",
      "   -1.9134578e+00 -1.9679726e+00]\n",
      "  [ 9.2118244e+00 -2.1657972e+00 -2.0584733e+00 ... -1.2250046e+00\n",
      "   -1.9401730e+00 -1.7737877e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.3108306e+00 -2.2582891e+00 -1.9954650e+00 ... -1.3321657e+00\n",
      "    5.3420097e-01  1.3868991e+00]\n",
      "  [-1.2517616e-01  6.1403787e-01 -3.7833450e+00 ... -2.6512072e+00\n",
      "    6.4982510e+00 -2.2371690e+00]\n",
      "  [ 2.3094063e-01 -2.6978276e+00 -1.8210014e+00 ... -4.9597445e-01\n",
      "    3.0108614e+00  2.5710776e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 7.8658003e-01 -3.0183271e-01 -8.5304344e-01 ... -2.6547667e-01\n",
      "   -3.8812238e-01 -1.2771020e-02]\n",
      "  [ 8.4109249e+00 -1.4914355e+00 -1.5835276e+00 ... -1.6558020e+00\n",
      "   -2.1149297e+00 -2.4469686e+00]\n",
      "  [ 8.3816738e+00 -1.7000188e+00 -1.1719019e+00 ... -1.2840710e+00\n",
      "   -2.3549178e+00 -2.2363238e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 3.5243821e+00 -2.0399644e+00 -2.2699170e+00 ... -1.4131519e+00\n",
      "    5.4198289e-01  1.9791265e-01]\n",
      "  [ 9.2977867e+00 -2.0998597e+00 -2.2765801e+00 ... -1.7311976e+00\n",
      "   -1.6597309e+00 -1.5474963e+00]\n",
      "  [ 9.3538733e+00 -2.1560354e+00 -2.3925934e+00 ... -1.6559576e+00\n",
      "   -1.3933301e+00 -1.5200186e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.9747071e+00 -1.7500048e+00 -2.1477189e+00 ... -1.3163379e+00\n",
      "    1.5073534e-02 -5.5394220e-01]\n",
      "  [ 9.3424072e+00 -1.9042218e+00 -2.1544986e+00 ... -1.5350696e+00\n",
      "   -1.9274206e+00 -1.6605744e+00]\n",
      "  [ 9.3732395e+00 -2.1027765e+00 -2.1425111e+00 ... -1.2314725e+00\n",
      "   -2.0007193e+00 -1.4612694e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.2227967e-01 -7.9023933e-01 -8.9657778e-01 ...  2.2979176e-02\n",
      "   -1.4039934e-01  1.3853985e+00]\n",
      "  [-3.5944889e+00  4.8931053e-01 -2.8391039e+00 ... -1.3275791e+00\n",
      "    6.1717343e+00 -7.4180621e-01]\n",
      "  [-2.1271687e+00 -3.8355486e+00  4.3963027e-01 ...  2.4281638e+00\n",
      "   -8.1624657e-01  6.2357426e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=2505, training_loss=0.18659199897400633, metrics={'train_runtime': 816.8133, 'train_samples_per_second': 73.53, 'train_steps_per_second': 3.067, 'total_flos': 1308863997899304.0, 'train_loss': 0.18659199897400633, 'epoch': 3.0})"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "training_args.logging_steps = len(corpora_encoded[\"train\"]) // batch_size\n",
    "training_args.output_dir = \"xlm-roberta-base-finetuned-panx-all\"\n",
    "\n",
    "trainer = Trainer(model_init=model_init, args=training_args,\n",
    "    data_collator=data_collator, compute_metrics=compute_metrics,\n",
    "    tokenizer=xlmr_tokenizer, train_dataset=corpora_encoded[\"train\"],\n",
    "    eval_dataset=corpora_encoded[\"validation\"])\n",
    "\n",
    "trainer.train()\n",
    "# trainer.push_to_hub(commit_message=\"Training completed!\") # PAUL - COMMENTING OUT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "bvruJiS63pUz",
    "outputId": "bf8850dc-d9de-449f-e7e7-d01996a50409"
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 7.9403543e-01 -1.2328169e+00 -1.1343009e+00 ... -3.5954663e-01\n",
      "    5.5029955e-02  1.7563232e+00]\n",
      "  [ 8.7740116e+00 -1.9510223e+00 -2.0150390e+00 ... -1.7945001e+00\n",
      "   -1.7943008e+00 -1.9372224e+00]\n",
      "  [ 8.7262363e+00 -1.8768736e+00 -2.0282381e+00 ... -1.7946838e+00\n",
      "   -1.7372109e+00 -1.9021082e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.1357555e+00 -2.4435581e-01 -4.9560294e-01 ...  2.8678373e-01\n",
      "   -1.8097545e+00 -2.4668186e+00]\n",
      "  [ 9.4032402e+00 -1.8462368e+00 -2.1455066e+00 ... -1.5149575e+00\n",
      "   -1.9847606e+00 -1.7469230e+00]\n",
      "  [-2.5444090e+00  7.4705591e+00 -1.1881567e+00 ... -1.2337635e+00\n",
      "   -7.3743641e-01 -3.8617780e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.5492878e+00 -1.7722182e-01 -6.5001714e-01 ... -3.4214124e-01\n",
      "   -8.3117306e-01 -7.6000893e-01]\n",
      "  [ 8.3047752e+00 -1.7030437e+00 -1.9773284e+00 ... -1.2101864e+00\n",
      "   -1.8968428e+00 -2.0921550e+00]\n",
      "  [ 7.7719064e+00 -1.4420452e+00 -1.8129889e+00 ... -1.0774573e+00\n",
      "   -1.9411168e+00 -1.9962401e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 3.7049079e+00 -1.7058775e+00 -1.7719470e+00 ...  7.8778845e-01\n",
      "   -9.6372205e-01 -1.8462728e+00]\n",
      "  [ 9.1445665e+00 -1.6731716e+00 -2.2847228e+00 ... -1.4712863e+00\n",
      "   -1.5622869e+00 -2.2224197e+00]\n",
      "  [ 9.1500130e+00 -1.8595370e+00 -2.3873963e+00 ... -1.6946853e+00\n",
      "   -1.5241489e+00 -1.9189661e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 4.0592670e+00 -1.8238076e+00 -1.9059885e+00 ... -7.9785752e-01\n",
      "   -4.7403282e-01 -1.5834234e+00]\n",
      "  [ 9.3297701e+00 -1.8454094e+00 -2.0703259e+00 ... -1.7976297e+00\n",
      "   -1.9174297e+00 -1.8046637e+00]\n",
      "  [ 7.1158457e+00 -1.6422569e+00 -3.2324183e+00 ... -2.0789015e+00\n",
      "    5.6484085e-01 -1.6180109e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.0894614e+00 -1.2535007e+00 -1.4752293e+00 ... -7.8295696e-01\n",
      "    5.4107428e-01  1.1545080e+00]\n",
      "  [-2.5435162e+00 -7.0674801e-01 -3.1990604e+00 ... -1.5869755e+00\n",
      "    7.3057847e+00 -5.5959821e-01]\n",
      "  [-1.3964159e+00 -3.8920701e+00 -8.2765067e-01 ...  5.0073445e-01\n",
      "    9.8578435e-01  6.6018515e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 1.1433227e+00  3.3091167e-01  1.0137870e-01 ... -3.0032197e-01\n",
      "   -1.2668253e+00 -1.2383301e+00]\n",
      "  [ 8.4667873e+00 -1.5044104e+00 -1.6150807e+00 ... -1.6498197e+00\n",
      "   -2.1245172e+00 -2.4466200e+00]\n",
      "  [ 8.4561510e+00 -1.7117666e+00 -1.1377261e+00 ... -1.2974257e+00\n",
      "   -2.3000474e+00 -2.3163204e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.6842725e+00 -9.3407941e-01 -1.5313141e+00 ... -9.4526792e-01\n",
      "    2.6799738e-01 -7.2772004e-02]\n",
      "  [ 7.5703006e+00 -9.7942120e-01 -2.7906024e+00 ... -2.2422421e+00\n",
      "   -4.0735090e-01 -2.5636103e+00]\n",
      "  [ 7.0775766e+00 -1.6942604e+00 -2.4678667e+00 ... -1.8710165e+00\n",
      "   -3.8173041e-01 -2.1424766e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 3.7944582e+00  1.2545741e-01  2.2553067e-01 ... -1.3081478e+00\n",
      "   -1.7927846e+00 -2.8680077e+00]\n",
      "  [ 9.2683010e+00 -1.7391561e+00 -1.9451324e+00 ... -1.7212313e+00\n",
      "   -2.1118138e+00 -1.9500804e+00]\n",
      "  [ 8.3074255e+00 -6.5727144e-01 -2.7279055e+00 ... -2.1105726e+00\n",
      "   -1.5171323e+00 -2.5453951e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 1.6953656e-01 -7.9049939e-01 -8.9389527e-01 ...  1.6159400e+00\n",
      "   -9.5686668e-01  1.6505532e-01]\n",
      "  [ 8.3203526e+00 -1.8967669e+00 -1.2429874e+00 ... -8.5795403e-01\n",
      "   -2.6061015e+00 -1.8765409e+00]\n",
      "  [-1.1461931e+00  4.5783082e-01 -3.5780873e+00 ... -1.4585732e+00\n",
      "    2.8417873e+00 -2.9569800e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.0035565e+00 -7.5910276e-01 -7.4239409e-01 ...  2.0560431e+00\n",
      "   -1.7361887e+00 -1.6027472e+00]\n",
      "  [-1.1495358e+00  1.4213479e+00 -3.3995943e+00 ... -1.7307311e+00\n",
      "    1.7934287e+00 -3.9077861e+00]\n",
      "  [-1.3899777e+00 -2.6538594e+00  1.9292106e-01 ...  6.1102128e+00\n",
      "   -2.6071072e+00 -7.1241295e-01]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]\n",
      "\n",
      " [[ 1.4886800e+00  4.5068672e-01  4.5812282e-01 ... -5.6036490e-01\n",
      "   -1.4457352e+00 -1.8573258e+00]\n",
      "  [ 8.5577135e+00 -1.5336685e+00 -1.6364994e+00 ... -1.6372572e+00\n",
      "   -2.2867522e+00 -2.3422248e+00]\n",
      "  [ 8.6189871e+00 -1.6546586e+00 -1.2657082e+00 ... -1.3793399e+00\n",
      "   -2.4146471e+00 -2.3188896e+00]\n",
      "  ...\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]\n",
      "  [-1.0000000e+02 -1.0000000e+02 -1.0000000e+02 ... -1.0000000e+02\n",
      "   -1.0000000e+02 -1.0000000e+02]]]\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 2.37664986e+00 -1.18980038e+00 -1.40319681e+00 ...  2.69460589e-01\n",
      "   -9.57246840e-01 -3.41033250e-01]\n",
      "  [ 9.00279617e+00 -2.05867219e+00 -2.21058798e+00 ... -1.87437618e+00\n",
      "   -1.38784897e+00 -1.75782955e+00]\n",
      "  [ 8.93927097e+00 -2.26200199e+00 -2.39393759e+00 ... -1.67596889e+00\n",
      "   -1.32777250e+00 -1.58036625e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 4.60199785e+00 -2.52526164e+00 -2.59831595e+00 ... -1.54326129e+00\n",
      "    4.21657205e-01  5.60146153e-01]\n",
      "  [ 8.98218632e+00 -2.02827954e+00 -2.34604955e+00 ... -1.68379533e+00\n",
      "   -1.47733700e+00 -1.75530493e+00]\n",
      "  [ 9.20008945e+00 -2.32323790e+00 -2.09513474e+00 ... -1.52083611e+00\n",
      "   -1.93531907e+00 -1.27233791e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 8.68442878e-02 -3.36237222e-01 -5.77773273e-01 ...  1.71678162e+00\n",
      "   -1.16072249e+00 -2.05935482e-02]\n",
      "  [-1.80251598e+00  1.85268998e+00 -2.88267112e+00 ... -7.63100743e-01\n",
      "    1.36668158e+00 -3.14969301e+00]\n",
      "  [-2.32194448e+00 -2.60599494e+00  1.72250581e+00 ...  5.71251392e+00\n",
      "   -2.56446600e+00  1.15910280e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 1.29483986e+00  1.09482974e-01  7.92726696e-01 ... -4.28861886e-01\n",
      "   -1.46312547e+00 -1.16338348e+00]\n",
      "  [ 7.57433605e+00 -1.63630450e+00 -2.19786382e+00 ... -1.38066399e+00\n",
      "   -1.28476107e+00 -2.29857731e+00]\n",
      "  [ 7.31281567e+00 -2.38192582e+00 -1.51705182e+00 ... -1.30068243e+00\n",
      "   -1.94840312e+00 -1.64441729e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 5.64824581e-01 -5.44498205e-01 -7.38820612e-01 ...  1.15763843e+00\n",
      "   -1.05834937e+00 -3.99090230e-01]\n",
      "  [-5.56691825e-01  4.30008560e-01 -3.63569641e+00 ... -1.87127030e+00\n",
      "    1.03925681e+00 -3.48473954e+00]\n",
      "  [-2.44333580e-01 -2.32457757e+00 -1.73685658e+00 ...  3.83891606e+00\n",
      "   -1.78962827e+00 -1.68076241e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[-1.31721795e+00 -4.14665997e-01  8.43077719e-01 ...  3.55636024e+00\n",
      "   -2.24033594e+00 -7.20406950e-01]\n",
      "  [-1.89073563e+00  2.20637226e+00 -2.61227655e+00 ... -1.30796218e+00\n",
      "    1.07297623e+00 -3.86474276e+00]\n",
      "  [-1.53574240e+00 -2.96734834e+00  2.54316211e+00 ...  6.04994249e+00\n",
      "   -3.18458939e+00  7.71829665e-01]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]]\n"
     ]
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_pred.predictions  [[[ 4.99528790e+00 -1.99973273e+00 -1.99247622e+00 ... -7.10512519e-01\n",
      "   -7.69161880e-01 -1.96727896e+00]\n",
      "  [ 9.37466335e+00 -2.13861918e+00 -1.93970203e+00 ... -1.47846460e+00\n",
      "   -2.12122059e+00 -1.59690106e+00]\n",
      "  [ 9.30642986e+00 -2.35838914e+00 -1.97116661e+00 ... -1.43320179e+00\n",
      "   -2.03499126e+00 -1.36883056e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 1.74531853e+00  6.10272765e-01  1.31297600e+00 ... -1.02340531e+00\n",
      "   -1.61554837e+00 -2.34866476e+00]\n",
      "  [-1.50250018e+00  8.09914494e+00 -1.76422799e+00 ... -1.94632566e+00\n",
      "   -7.75494874e-01 -3.79209542e+00]\n",
      "  [-1.88332856e+00 -9.78822529e-01  7.54941988e+00 ...  1.36956024e+00\n",
      "   -3.07306957e+00 -1.42559600e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 1.11030988e-01 -5.08746982e-01 -6.09975696e-01 ...  2.28693828e-01\n",
      "   -4.53811258e-01  9.79507506e-01]\n",
      "  [-1.88922083e+00  2.13256574e+00 -3.18206286e+00 ... -2.19673038e+00\n",
      "    5.02007008e+00 -2.84153509e+00]\n",
      "  [-1.99556446e+00 -2.00211573e+00  5.71012735e-01 ...  1.28829670e+00\n",
      "   -1.70065224e-01  3.25760770e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " ...\n",
      "\n",
      " [[ 2.98530519e-01  1.06485999e+00  2.48090339e+00 ... -1.01703666e-01\n",
      "   -2.01572299e+00 -2.06392884e+00]\n",
      "  [-2.02760267e+00  7.81021786e+00 -1.11618543e+00 ... -1.61482108e+00\n",
      "   -1.28439867e+00 -3.61262846e+00]\n",
      "  [-1.41687810e+00 -9.60719943e-01  7.43542910e+00 ...  1.40002310e+00\n",
      "   -3.22659302e+00 -1.47916412e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 9.82634068e-01 -6.95255220e-01 -9.55693841e-01 ... -3.54314521e-02\n",
      "   -4.33846265e-01  2.66657889e-01]\n",
      "  [ 8.61083889e+00 -1.74175763e+00 -1.65848351e+00 ... -1.48435724e+00\n",
      "   -2.30050135e+00 -2.18389201e+00]\n",
      "  [ 8.47810364e+00 -1.41284180e+00 -1.85885823e+00 ... -1.68097925e+00\n",
      "   -1.84837961e+00 -2.30352902e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]\n",
      "\n",
      " [[ 3.39996740e-02 -1.61012352e+00 -1.08897316e+00 ...  2.61358929e+00\n",
      "    3.17936540e-01 -1.36344314e-01]\n",
      "  [-8.25844586e-01  2.03731909e-01 -3.47547507e+00 ... -6.87000215e-01\n",
      "    3.12044764e+00 -2.80614257e+00]\n",
      "  [-3.89074296e-01 -3.39344430e+00  3.89304943e-02 ...  4.85991621e+00\n",
      "   -1.21272695e+00  1.66818666e+00]\n",
      "  ...\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]\n",
      "  [-1.00000000e+02 -1.00000000e+02 -1.00000000e+02 ... -1.00000000e+02\n",
      "   -1.00000000e+02 -1.00000000e+02]]]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "for idx, lang in enumerate(languages):\n",
    "    f1_scores[\"all\"][lang] = get_f1_score(trainer, corpora[idx][\"test\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 175
    },
    "id": "JLr0m5Kk3pUz",
    "outputId": "cce13fae-9d33-4130-960a-d68d2c370951"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-d86d82a6-9d4d-4368-9560-25c0d008f184\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Evaluated on</th>\n",
       "      <th>de</th>\n",
       "      <th>fr</th>\n",
       "      <th>it</th>\n",
       "      <th>en</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fine-tune on</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>de</th>\n",
       "      <td>0.8691</td>\n",
       "      <td>0.7016</td>\n",
       "      <td>0.6739</td>\n",
       "      <td>0.5921</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>each</th>\n",
       "      <td>0.8691</td>\n",
       "      <td>0.8372</td>\n",
       "      <td>0.8178</td>\n",
       "      <td>0.7084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>all</th>\n",
       "      <td>0.8697</td>\n",
       "      <td>0.8729</td>\n",
       "      <td>0.8698</td>\n",
       "      <td>0.7724</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d86d82a6-9d4d-4368-9560-25c0d008f184')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-d86d82a6-9d4d-4368-9560-25c0d008f184 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-d86d82a6-9d4d-4368-9560-25c0d008f184');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "Evaluated on      de      fr      it      en\n",
       "Fine-tune on                                \n",
       "de            0.8691  0.7016  0.6739  0.5921\n",
       "each          0.8691  0.8372  0.8178  0.7084\n",
       "all           0.8697  0.8729  0.8698  0.7724"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores_data = {\"de\": f1_scores[\"de\"],\n",
    "               \"each\": {lang: f1_scores[lang][lang] for lang in languages},\n",
    "               \"all\": f1_scores[\"all\"]}\n",
    "f1_scores_df = pd.DataFrame(scores_data).T.round(4)\n",
    "f1_scores_df.rename_axis(index=\"Fine-tune on\", columns=\"Evaluated on\",\n",
    "                         inplace=True)\n",
    "f1_scores_df"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "V3_SHORTENED-for-YT_MW_4_multilingual-ner.ipynb",
   "provenance": []
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3.9.13 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "36cf16204b8548560b1c020c4e8fb5b57f0e4c58016f52f2d4be01e192833930"
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "01740edabff446a0935dd097dbf0f295": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "03a260f79a0e4f32b46698d959f22078": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "04d740ecc336428abf0cf4a790b9ffbd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4a2e03065d1d4ee28f85385299827c0a",
      "placeholder": "​",
      "style": "IPY_MODEL_600b1c7ca7864e419edfd33e6a702ab9",
      "value": " 3/3 [00:00&lt;00:00, 69.13it/s]"
     }
    },
    "0594d6b5cecc4b01bdbefdb36a329995": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_06101367c25445098881422afb123580",
      "placeholder": "​",
      "style": "IPY_MODEL_7cad496137b04ef2acd1c84f5162a0a8",
      "value": "100%"
     }
    },
    "060a8a505aab4fe7885611e30b6bbe37": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_83ac07d90ca94345aa20146b0a9c3edf",
      "placeholder": "​",
      "style": "IPY_MODEL_f1163b0f9eea40f8b086b18eda7b1e87",
      "value": "100%"
     }
    },
    "06101367c25445098881422afb123580": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0a95843a9a284cc2990947038bd17bf0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "0dfd91a2f6a8481b87bddd642feb3e06": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "11ec68d5eb0544c797fd95b73555d57e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "11f7b37d0b4d4939a7a1ecec4a411bb1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_de7d70c071094a678fe89ba73fcdf442",
      "placeholder": "​",
      "style": "IPY_MODEL_56baee0a1ce94b2c820c271adf32b801",
      "value": "100%"
     }
    },
    "12b5cb91b9a645e588235d3573cf7b31": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "134a5f727b8545d0b2442f77ba19ba85": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "154bdbef9b884e4084667340291dc3d3": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1565cda3d181427f8f2d38d6316a0e34": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "16c60bd460c949bc97c9820c2c0d971e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "176379cc286e4caabcdee0ae540b9f34": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_84f6e54498d545bb9d41df8f75aaf8c2",
       "IPY_MODEL_cfa6b5b58c8949e2b58a749269499fb7",
       "IPY_MODEL_e3096f901a644a439634bf609faded4f"
      ],
      "layout": "IPY_MODEL_f759ed05d93942899a1852af8c308824"
     }
    },
    "1a51eb4e60bd495b8f457533b43032e4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1e7c1b4e2a184835a55a9f6926399d2a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "1e9ccb7fc6cb40f8998e53666e5f720b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_6c558c28973d4d659fc9a4c7b01c9279",
       "IPY_MODEL_f9ce323e0969454cb4facbf6f1246c2f",
       "IPY_MODEL_f33d5356927141fabd5b6f2296eb3c15"
      ],
      "layout": "IPY_MODEL_e314843794f6448492bc6248131bb54b"
     }
    },
    "20d8e5de43b54c2690161d869571444d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "2128c0ba61d8423a9d92503defbed462": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a29fbee89c4b4289a3323d4b1fda6293",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_56596bff7af94bb8b78e027b5d347e3d",
      "value": 1
     }
    },
    "21dccd4f0cd54ba180fd6e5cccf4464a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "227468d815d049bfa9d9b11dd1d502de": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "24977036948944e3960e0f7654bb9150": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "25d86f7f215744d09a83e3986fcf3c20": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "273adb7bced045bd8321846e07746ab0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "276fb4f0480046b69b296d034af6a11f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f7094e82152a4f988e9a51612ddf57a3",
      "placeholder": "​",
      "style": "IPY_MODEL_b9b5e45f30fc48778aa6f2e0298652e4",
      "value": "100%"
     }
    },
    "286792edbe7b412e95a49912378213b8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1a51eb4e60bd495b8f457533b43032e4",
      "placeholder": "​",
      "style": "IPY_MODEL_80d299714ef04cd9801338d64af029f9",
      "value": " 1/1 [00:00&lt;00:00,  7.38ba/s]"
     }
    },
    "289a2a99024741808313f44980ee2efc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_25d86f7f215744d09a83e3986fcf3c20",
      "placeholder": "​",
      "style": "IPY_MODEL_8b778bacf758448385f5d3c7dbe502f7",
      "value": "100%"
     }
    },
    "299b8e07b33741e79ca4ec8d4e3b5f9f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2e746d5c0053487bab87f2d02b164d9f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3157930ac9eb4b35bb2e72e54bd4410a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "32223b0d3b624f78acbfbfb97f6917db": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_de214be9ed654a5f83a9bd083b4deba6",
      "max": 5,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_d69991d3c44f41ef988090bddec7bbc4",
      "value": 5
     }
    },
    "3312fae1e90c42d4929c95e2b90e669b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "33e629ed2e674752a91b2bfd4872992e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_154bdbef9b884e4084667340291dc3d3",
      "placeholder": "​",
      "style": "IPY_MODEL_2e746d5c0053487bab87f2d02b164d9f",
      "value": "100%"
     }
    },
    "364ac5a6b4d146eb8239c608e8c76fd2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "38c0b45e3ab448df8d8eef15a703d26c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d2f04c99f8384f9ebeb486b668c2881b",
      "placeholder": "​",
      "style": "IPY_MODEL_b52c73cb5a034b388682d92ada5ec67e",
      "value": " 5/5 [00:00&lt;00:00, 11.88ba/s]"
     }
    },
    "3a893b9431744117ac9b7e50c2a9f013": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_622cab19793f4f92910f10b755d30e52",
       "IPY_MODEL_90f1a2b774fe4c8d8c795b1b76f5fce2",
       "IPY_MODEL_ed920a6c4e8c4147a6f2ed98e08d64bb"
      ],
      "layout": "IPY_MODEL_299b8e07b33741e79ca4ec8d4e3b5f9f"
     }
    },
    "3b596911fa7a4821b62de803328477d7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5535ea8576404174b17cb4773c14112e",
      "placeholder": "​",
      "style": "IPY_MODEL_4a94340d8b084cc6be922f77465a64e9",
      "value": " 2/2 [00:00&lt;00:00,  6.47ba/s]"
     }
    },
    "3ce86a1149b14240a998541d525c41f2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3d5649e2ee0549c1a1a576b47bb2b2ad": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3ddaced9ad7b4ad3b9fc9629e8d0e240": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3fffcbd9bd684b8bbcb58d127b7b98ca": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "423d5c3212154705a1fd85851515c9f0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "430d63c852b941ba9a9174744d865d8a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4435c65b37fd483ca6fd6df94381f010": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "448a446fa3c34b73b9335c9e831a8e5e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_dfe170856aad4b0cb8ab484c1bb9044d",
       "IPY_MODEL_5e2aa9f014ac46e8993fa740ddeecdc5",
       "IPY_MODEL_87a442a80a18410aa1a8bd8b9a4fd763"
      ],
      "layout": "IPY_MODEL_80dfcdb2f016497f83055781655c3a4e"
     }
    },
    "4524099fda054038a5bef33ca175dd14": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "45f2ccb89613456a87b4f0b17e444f49": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "47864d588d8e40e6a5127b4c1ca8e9da": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "49bc2c3640174600b8f7023d5d97db6c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ccc243f8298e40b78cb90f56f4ab5f3e",
      "placeholder": "​",
      "style": "IPY_MODEL_24977036948944e3960e0f7654bb9150",
      "value": " 3/3 [00:00&lt;00:00, 69.86it/s]"
     }
    },
    "49bf43c1b30c4dbda112572d88d3ae89": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "4a2e03065d1d4ee28f85385299827c0a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4a94340d8b084cc6be922f77465a64e9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4aa1b456d232482c8b8ae0f090c8eee4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3312fae1e90c42d4929c95e2b90e669b",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_ca82a302c2a14c9aaea2ef3a8185777f",
      "value": 1
     }
    },
    "4afd7d35ce6e421dbc4fe67a2e59b061": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4b0e2338976d4ababfb13410b1a77a43": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4d5747d0052b48dfab6794ebdb1930b0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4ea3cd5b83024857a882f59b4a2ffe92": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4f448e3ebd8741548983425ed380ed14": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_81bf416562244225aa988dbe0b3b1bc5",
      "max": 3,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_20d8e5de43b54c2690161d869571444d",
      "value": 3
     }
    },
    "4fa2f7d96558420eaadfc8c9bbdd0119": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "5302b322abee472c8585944114d445ad": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "55022f8c3a7649ceb6d335a7d830bd28": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5535ea8576404174b17cb4773c14112e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "56596bff7af94bb8b78e027b5d347e3d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "5659cde4a272492fb14dda55400d6ef5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "56baee0a1ce94b2c820c271adf32b801": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "580c3d39bc774beda69eace3e4e00cdf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5c87ecea200e43e5b757469cf8a62d3c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9c1534cf995e47a5a31f1ede6f4e2926",
      "placeholder": "​",
      "style": "IPY_MODEL_74e18ecf7cea46b18feab1704c01b6cb",
      "value": "100%"
     }
    },
    "5df28d2528834a66a1e9e4962dd8b31f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_913b9d3327f24e2fb3f63ca8fc975572",
      "max": 2,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_134a5f727b8545d0b2442f77ba19ba85",
      "value": 2
     }
    },
    "5e2aa9f014ac46e8993fa740ddeecdc5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5659cde4a272492fb14dda55400d6ef5",
      "max": 3,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_3157930ac9eb4b35bb2e72e54bd4410a",
      "value": 3
     }
    },
    "5e7fa665ba56418ca351e9d049901aa4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "600b1c7ca7864e419edfd33e6a702ab9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "622cab19793f4f92910f10b755d30e52": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_03a260f79a0e4f32b46698d959f22078",
      "placeholder": "​",
      "style": "IPY_MODEL_d0e33ec011824dd6a25f49efa7a22fe6",
      "value": "100%"
     }
    },
    "648a592e0eeb4ee4b57365b5dcf8c703": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d47976209d6f49d2ac596fda9e926393",
      "placeholder": "​",
      "style": "IPY_MODEL_3ddaced9ad7b4ad3b9fc9629e8d0e240",
      "value": " 1/1 [00:00&lt;00:00,  7.04ba/s]"
     }
    },
    "6685a46a80f046d7a7108dc3812573f4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "675d8eebafc84f2093659df4a93f0180": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "67f229dbd1e449069eec44e0be7bf048": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6913c066eefa4d769d556d9c6403b634": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6a076dbd5f8b4009b4b8be6b00adb37c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f5837d6a41224c2ea7c165d4e6cd0f49",
      "placeholder": "​",
      "style": "IPY_MODEL_a89869b1ca3647f1995eb084a55d822b",
      "value": "100%"
     }
    },
    "6a3ebada114f49879e650cb890e713ea": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6c558c28973d4d659fc9a4c7b01c9279": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f0fd1897d30d467b999076fb6da052ee",
      "placeholder": "​",
      "style": "IPY_MODEL_b82f3962015b4b479398f7869417a198",
      "value": "100%"
     }
    },
    "6ccf7b324e3c400eaa454c0c11a3d04c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "6ded3cca879246e7a286ab3e47ddc518": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7222b506812340e3ba3bf8f9d76042d0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4d5747d0052b48dfab6794ebdb1930b0",
      "max": 3,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_be8698ef584c4c7cb7a99cd7e4c8d959",
      "value": 3
     }
    },
    "727ee0ced86b4cc996256a513e7961de": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7294bc8dcff24989a8ec684eb095063a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_787a3ead23584012a93a053fdf678bb2",
      "max": 3,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_1e7c1b4e2a184835a55a9f6926399d2a",
      "value": 3
     }
    },
    "74e18ecf7cea46b18feab1704c01b6cb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "75e551bbe19941079a3cb8e10bd2bd92": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "77512d0ba25844488753e322cc121907": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_423d5c3212154705a1fd85851515c9f0",
      "placeholder": "​",
      "style": "IPY_MODEL_9129785f192e46179d7a8faaa30b7f0a",
      "value": " 197/197 [00:17&lt;00:00, 11.43ba/s]"
     }
    },
    "785311ad23c14f2dbed096a9b054813f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4b0e2338976d4ababfb13410b1a77a43",
      "placeholder": "​",
      "style": "IPY_MODEL_580c3d39bc774beda69eace3e4e00cdf",
      "value": "100%"
     }
    },
    "7854caba960e425e9518a172af7f96a7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "787a3ead23584012a93a053fdf678bb2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7a5419c120804ae9bc3618f6c5dea22d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_dd6e54ead891406ebbbd20a308fb6e56",
      "max": 13,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_75e551bbe19941079a3cb8e10bd2bd92",
      "value": 13
     }
    },
    "7cad496137b04ef2acd1c84f5162a0a8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7db222dc3fb9457bae8daf4937977699": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "80d299714ef04cd9801338d64af029f9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "80dfcdb2f016497f83055781655c3a4e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "80e3d8e2a3bb47ddbf76225dfdeb03cc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4524099fda054038a5bef33ca175dd14",
      "placeholder": "​",
      "style": "IPY_MODEL_b595a25e58b04a459e6ea3ddeffba9dc",
      "value": " 3/3 [00:00&lt;00:00, 69.82it/s]"
     }
    },
    "815b038be78b446988e4a3af396e11d1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "81bf416562244225aa988dbe0b3b1bc5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "82e317f3fb1045adbb37bf162ce0e510": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_785311ad23c14f2dbed096a9b054813f",
       "IPY_MODEL_b9f7a0863a3045d39c328bc144bec0f8",
       "IPY_MODEL_cd1d3d5dd7a54b20be5130e60ce595e7"
      ],
      "layout": "IPY_MODEL_ca989bbad49c4865b2b8b8987a139c42"
     }
    },
    "83ac07d90ca94345aa20146b0a9c3edf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "84759f2e60c14548b285c0b32c39230b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_8a546e75a87245309f7a56eef8a68d9a",
       "IPY_MODEL_32223b0d3b624f78acbfbfb97f6917db",
       "IPY_MODEL_38c0b45e3ab448df8d8eef15a703d26c"
      ],
      "layout": "IPY_MODEL_6913c066eefa4d769d556d9c6403b634"
     }
    },
    "84f6e54498d545bb9d41df8f75aaf8c2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_45f2ccb89613456a87b4f0b17e444f49",
      "placeholder": "​",
      "style": "IPY_MODEL_b62fb85bcd9846f391016e02911597d8",
      "value": "100%"
     }
    },
    "87a442a80a18410aa1a8bd8b9a4fd763": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ab4d3e6b79234a6a904e78425868c199",
      "placeholder": "​",
      "style": "IPY_MODEL_c7671fe58c79492dae2ec41dca7c6d9b",
      "value": " 3/3 [00:00&lt;00:00, 12.69ba/s]"
     }
    },
    "889691d5397a4e07acbbc3c3b6db7561": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_55022f8c3a7649ceb6d335a7d830bd28",
      "max": 197,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_1565cda3d181427f8f2d38d6316a0e34",
      "value": 197
     }
    },
    "8a546e75a87245309f7a56eef8a68d9a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4ea3cd5b83024857a882f59b4a2ffe92",
      "placeholder": "​",
      "style": "IPY_MODEL_67f229dbd1e449069eec44e0be7bf048",
      "value": "100%"
     }
    },
    "8b778bacf758448385f5d3c7dbe502f7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8cd59a5a34d148d59b2c8fef23ac69b1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8e1b38886d4843988814fc8f66953808": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_b2202c1db1d94ae68de1e7dd25b652b8",
       "IPY_MODEL_f719f7fdcc894ceea99ec78d7b84fbb7",
       "IPY_MODEL_e56882ac1c034cb49be257285fc1bf9d"
      ],
      "layout": "IPY_MODEL_c306ab035b4e44a8bc1f1c813b224f74"
     }
    },
    "90f1a2b774fe4c8d8c795b1b76f5fce2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6ded3cca879246e7a286ab3e47ddc518",
      "max": 2,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_5302b322abee472c8585944114d445ad",
      "value": 2
     }
    },
    "9129785f192e46179d7a8faaa30b7f0a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "913b9d3327f24e2fb3f63ca8fc975572": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "93de5543aba84405a952dee2aa257508": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a468093f9adf410596d7ca2dd880f05c",
       "IPY_MODEL_2128c0ba61d8423a9d92503defbed462",
       "IPY_MODEL_df2eecd3afb743d3bdbe58b8a1c268a9"
      ],
      "layout": "IPY_MODEL_3d5649e2ee0549c1a1a576b47bb2b2ad"
     }
    },
    "9b3fca254ee24be997161ff32ed41910": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_0594d6b5cecc4b01bdbefdb36a329995",
       "IPY_MODEL_7294bc8dcff24989a8ec684eb095063a",
       "IPY_MODEL_ae54f863ffee49a0a29a0133f36a3e71"
      ],
      "layout": "IPY_MODEL_815b038be78b446988e4a3af396e11d1"
     }
    },
    "9c1534cf995e47a5a31f1ede6f4e2926": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9e6f411b91be4631970f9f92116f29e8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a29fbee89c4b4289a3323d4b1fda6293": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a468093f9adf410596d7ca2dd880f05c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c747523f9e0b473a9cacbd10ca7d5d16",
      "placeholder": "​",
      "style": "IPY_MODEL_8cd59a5a34d148d59b2c8fef23ac69b1",
      "value": "100%"
     }
    },
    "a6bbcc9b57094e988c4c44eec6a720f4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a6f991c86951454f853978e08d81f36c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_5c87ecea200e43e5b757469cf8a62d3c",
       "IPY_MODEL_5df28d2528834a66a1e9e4962dd8b31f",
       "IPY_MODEL_3b596911fa7a4821b62de803328477d7"
      ],
      "layout": "IPY_MODEL_ba197b6682044734b5a4ccd92a78c945"
     }
    },
    "a7fef51748b7414796602bf7d2bc3ed6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "a87b0705985046cf9abb1e906d7d196b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0dfd91a2f6a8481b87bddd642feb3e06",
      "placeholder": "​",
      "style": "IPY_MODEL_0a95843a9a284cc2990947038bd17bf0",
      "value": "100%"
     }
    },
    "a89869b1ca3647f1995eb084a55d822b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ab4d3e6b79234a6a904e78425868c199": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ae54f863ffee49a0a29a0133f36a3e71": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b57cef29cc2948f8803504790aed8a16",
      "placeholder": "​",
      "style": "IPY_MODEL_f5045a19461e4740a5a0848561753c14",
      "value": " 3/3 [00:00&lt;00:00, 60.94it/s]"
     }
    },
    "ae5a2a816a654ff79f40ed0c21dd86ba": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "af02475f27bb4c24913a7050d6fe61d2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b2202c1db1d94ae68de1e7dd25b652b8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_12b5cb91b9a645e588235d3573cf7b31",
      "placeholder": "​",
      "style": "IPY_MODEL_01740edabff446a0935dd097dbf0f295",
      "value": "100%"
     }
    },
    "b27bcf76e53f4035ad0464bcf0fe490f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b52c73cb5a034b388682d92ada5ec67e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b57cef29cc2948f8803504790aed8a16": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b595a25e58b04a459e6ea3ddeffba9dc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b62fb85bcd9846f391016e02911597d8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b82f3962015b4b479398f7869417a198": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b9b5e45f30fc48778aa6f2e0298652e4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b9f7a0863a3045d39c328bc144bec0f8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_364ac5a6b4d146eb8239c608e8c76fd2",
      "max": 3,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_49bf43c1b30c4dbda112572d88d3ae89",
      "value": 3
     }
    },
    "ba197b6682044734b5a4ccd92a78c945": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bb8e24c80a1a4b38bf11f2b4ee00e92f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bdf37dd464d24f8da6212c560fcc4ddc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_276fb4f0480046b69b296d034af6a11f",
       "IPY_MODEL_fa21c19946574d4db8be3a94708685a7",
       "IPY_MODEL_c4558d4453124b539d6dae2d77e0cc22"
      ],
      "layout": "IPY_MODEL_6a3ebada114f49879e650cb890e713ea"
     }
    },
    "be8698ef584c4c7cb7a99cd7e4c8d959": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "c306ab035b4e44a8bc1f1c813b224f74": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c4558d4453124b539d6dae2d77e0cc22": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7db222dc3fb9457bae8daf4937977699",
      "placeholder": "​",
      "style": "IPY_MODEL_6685a46a80f046d7a7108dc3812573f4",
      "value": " 1/1 [00:00&lt;00:00,  9.37ba/s]"
     }
    },
    "c5247e0ac1164f479eac9caebdd8f830": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_33e629ed2e674752a91b2bfd4872992e",
       "IPY_MODEL_4f448e3ebd8741548983425ed380ed14",
       "IPY_MODEL_04d740ecc336428abf0cf4a790b9ffbd"
      ],
      "layout": "IPY_MODEL_727ee0ced86b4cc996256a513e7961de"
     }
    },
    "c5c5d60cb9794a68ae13371e6f76a1da": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c747523f9e0b473a9cacbd10ca7d5d16": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c7671fe58c79492dae2ec41dca7c6d9b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c78295b5fa504243b898cea0f510c3ed": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c8394ae833884a4ab89622c2597dd1ae": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ed76b36c51f443ca829c6538eba769d0",
       "IPY_MODEL_4aa1b456d232482c8b8ae0f090c8eee4",
       "IPY_MODEL_286792edbe7b412e95a49912378213b8"
      ],
      "layout": "IPY_MODEL_4435c65b37fd483ca6fd6df94381f010"
     }
    },
    "ca2f707159c543048e3eac987fff4578": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_6a076dbd5f8b4009b4b8be6b00adb37c",
       "IPY_MODEL_f86eb20bed6540efb3f931dfa142d7bc",
       "IPY_MODEL_49bc2c3640174600b8f7023d5d97db6c"
      ],
      "layout": "IPY_MODEL_9e6f411b91be4631970f9f92116f29e8"
     }
    },
    "ca82a302c2a14c9aaea2ef3a8185777f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "ca989bbad49c4865b2b8b8987a139c42": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "caf40dc96d5e460884a2aa71da7b1b0b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "cb7e6845f9024b91aa4fed9f463db6d2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b27bcf76e53f4035ad0464bcf0fe490f",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_16c60bd460c949bc97c9820c2c0d971e",
      "value": 1
     }
    },
    "ccc243f8298e40b78cb90f56f4ab5f3e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "cd1d3d5dd7a54b20be5130e60ce595e7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7854caba960e425e9518a172af7f96a7",
      "placeholder": "​",
      "style": "IPY_MODEL_11ec68d5eb0544c797fd95b73555d57e",
      "value": " 3/3 [00:00&lt;00:00, 85.60it/s]"
     }
    },
    "cfa6b5b58c8949e2b58a749269499fb7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_430d63c852b941ba9a9174744d865d8a",
      "max": 3,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_f55e19b106814d34ab74ba2e908aff2c",
      "value": 3
     }
    },
    "d0e33ec011824dd6a25f49efa7a22fe6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d2f04c99f8384f9ebeb486b668c2881b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d47976209d6f49d2ac596fda9e926393": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d69991d3c44f41ef988090bddec7bbc4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "d7e613964c8b4103b4b718df31da6f7b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_11f7b37d0b4d4939a7a1ecec4a411bb1",
       "IPY_MODEL_7222b506812340e3ba3bf8f9d76042d0",
       "IPY_MODEL_80e3d8e2a3bb47ddbf76225dfdeb03cc"
      ],
      "layout": "IPY_MODEL_caf40dc96d5e460884a2aa71da7b1b0b"
     }
    },
    "dd6e54ead891406ebbbd20a308fb6e56": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "de214be9ed654a5f83a9bd083b4deba6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "de7d70c071094a678fe89ba73fcdf442": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "df272f940b3341bdb5eb15b8084b11af": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_289a2a99024741808313f44980ee2efc",
       "IPY_MODEL_cb7e6845f9024b91aa4fed9f463db6d2",
       "IPY_MODEL_648a592e0eeb4ee4b57365b5dcf8c703"
      ],
      "layout": "IPY_MODEL_f26e946754a14424a5c686c3177e9ed8"
     }
    },
    "df2eecd3afb743d3bdbe58b8a1c268a9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f3e27114c3ef4c3c94dda99c0c04ef9f",
      "placeholder": "​",
      "style": "IPY_MODEL_a7fef51748b7414796602bf7d2bc3ed6",
      "value": " 1/1 [00:00&lt;00:00,  9.34ba/s]"
     }
    },
    "dfe170856aad4b0cb8ab484c1bb9044d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_47864d588d8e40e6a5127b4c1ca8e9da",
      "placeholder": "​",
      "style": "IPY_MODEL_c78295b5fa504243b898cea0f510c3ed",
      "value": "100%"
     }
    },
    "e1352047b2144c76aa4a7619677128e7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "e3096f901a644a439634bf609faded4f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fc29df0299814ab489deeb0e1fd9f89f",
      "placeholder": "​",
      "style": "IPY_MODEL_273adb7bced045bd8321846e07746ab0",
      "value": " 3/3 [00:00&lt;00:00, 12.62ba/s]"
     }
    },
    "e314843794f6448492bc6248131bb54b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e419c224a82b4ab4a09271dee9c22df2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e56882ac1c034cb49be257285fc1bf9d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ae5a2a816a654ff79f40ed0c21dd86ba",
      "placeholder": "​",
      "style": "IPY_MODEL_4afd7d35ce6e421dbc4fe67a2e59b061",
      "value": " 5/5 [00:00&lt;00:00, 11.38ba/s]"
     }
    },
    "eb07821864444733b71a1e21c5f167a6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ed76b36c51f443ca829c6538eba769d0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a6bbcc9b57094e988c4c44eec6a720f4",
      "placeholder": "​",
      "style": "IPY_MODEL_fb70739a2c8f40ca9a992851a47c64ec",
      "value": "100%"
     }
    },
    "ed920a6c4e8c4147a6f2ed98e08d64bb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_675d8eebafc84f2093659df4a93f0180",
      "placeholder": "​",
      "style": "IPY_MODEL_f71aa4ec35b346a684ff9f90bb7ea04d",
      "value": " 2/2 [00:00&lt;00:00,  6.24ba/s]"
     }
    },
    "edd5de97408a4760a2e770e49c189246": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a87b0705985046cf9abb1e906d7d196b",
       "IPY_MODEL_7a5419c120804ae9bc3618f6c5dea22d",
       "IPY_MODEL_ef43aed878ca459e96b039d44e4f1cbd"
      ],
      "layout": "IPY_MODEL_227468d815d049bfa9d9b11dd1d502de"
     }
    },
    "ef43aed878ca459e96b039d44e4f1cbd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_af02475f27bb4c24913a7050d6fe61d2",
      "placeholder": "​",
      "style": "IPY_MODEL_3ce86a1149b14240a998541d525c41f2",
      "value": " 13/13 [00:01&lt;00:00,  7.14ba/s]"
     }
    },
    "f0fd1897d30d467b999076fb6da052ee": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f1163b0f9eea40f8b086b18eda7b1e87": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f26e946754a14424a5c686c3177e9ed8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f33d5356927141fabd5b6f2296eb3c15": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c5c5d60cb9794a68ae13371e6f76a1da",
      "placeholder": "​",
      "style": "IPY_MODEL_eb07821864444733b71a1e21c5f167a6",
      "value": " 7/7 [00:01&lt;00:00,  4.34ba/s]"
     }
    },
    "f3e27114c3ef4c3c94dda99c0c04ef9f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f5045a19461e4740a5a0848561753c14": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f55e19b106814d34ab74ba2e908aff2c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "f5837d6a41224c2ea7c165d4e6cd0f49": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f636b121dd694106be6b7938f3bd677c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "f7094e82152a4f988e9a51612ddf57a3": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f719f7fdcc894ceea99ec78d7b84fbb7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5e7fa665ba56418ca351e9d049901aa4",
      "max": 5,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_f636b121dd694106be6b7938f3bd677c",
      "value": 5
     }
    },
    "f71aa4ec35b346a684ff9f90bb7ea04d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f759ed05d93942899a1852af8c308824": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f86eb20bed6540efb3f931dfa142d7bc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_bb8e24c80a1a4b38bf11f2b4ee00e92f",
      "max": 3,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_6ccf7b324e3c400eaa454c0c11a3d04c",
      "value": 3
     }
    },
    "f9ce323e0969454cb4facbf6f1246c2f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_21dccd4f0cd54ba180fd6e5cccf4464a",
      "max": 7,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_e1352047b2144c76aa4a7619677128e7",
      "value": 7
     }
    },
    "fa21c19946574d4db8be3a94708685a7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3fffcbd9bd684b8bbcb58d127b7b98ca",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_4fa2f7d96558420eaadfc8c9bbdd0119",
      "value": 1
     }
    },
    "fb70739a2c8f40ca9a992851a47c64ec": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "fc29df0299814ab489deeb0e1fd9f89f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fdb2de29b6dd4e6fbd4d790b97c8fd9c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_060a8a505aab4fe7885611e30b6bbe37",
       "IPY_MODEL_889691d5397a4e07acbbc3c3b6db7561",
       "IPY_MODEL_77512d0ba25844488753e322cc121907"
      ],
      "layout": "IPY_MODEL_e419c224a82b4ab4a09271dee9c22df2"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
